Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:59:05.759148 398 test_util.cc:268] Using random seed: 277340572
I20240503 01:59:05.765657 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:05.765750 398 ts_itest-base.cc:113] --------------
I20240503 01:59:05.765779 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 01:59:05.765805 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:05.765830 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:05Z Disabled control of system clock
I20240503 01:59:05.774498 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35171
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35171
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 01:59:05.873549 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:05.873741 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:05.873777 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:05.876626 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:05.876683 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:05.876706 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:05.876730 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:05.876747 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:05.880112 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35171
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:05.880829 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:05.881824 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:05.889199 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:05.889986 410 server_base.cc:1034] running on GCE node
W20240503 01:59:05.890676 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:05.891434 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:05.891908 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:05.892691 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:05.893839 410 hybrid_clock.cc:648] HybridClock initialized: now 1714701545893798 us; error 57 us; skew 500 ppm
I20240503 01:59:05.895512 410 webserver.cc:460] Webserver started at http://127.0.99.190:44497/ using document root <none> and password file <none>
I20240503 01:59:05.896056 410 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:05.896113 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:05.896277 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:05.897791 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0045eb859f304fbd9ffc8c7eda824b04"
format_stamp: "Formatted at 2024-05-03 01:59:05 on dist-test-slave-fvr5"
I20240503 01:59:05.898243 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0045eb859f304fbd9ffc8c7eda824b04"
format_stamp: "Formatted at 2024-05-03 01:59:05 on dist-test-slave-fvr5"
I20240503 01:59:05.901276 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 01:59:05.903201 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:05.904127 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:59:05.904235 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0045eb859f304fbd9ffc8c7eda824b04"
format_stamp: "Formatted at 2024-05-03 01:59:05 on dist-test-slave-fvr5"
I20240503 01:59:05.904310 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:05.918481 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:05.919001 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:05.919167 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:05.925261 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35171
I20240503 01:59:05.925278 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35171 every 8 connection(s)
I20240503 01:59:05.926128 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:05.928514 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:05.929075 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 410
I20240503 01:59:05.929221 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:05.934396 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: Bootstrap starting.
I20240503 01:59:05.936743 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:05.937438 477 log.cc:824] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:05.938995 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: No bootstrap required, opened a new log
I20240503 01:59:05.941596 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } }
I20240503 01:59:05.941807 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:05.941893 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0045eb859f304fbd9ffc8c7eda824b04, State: Initialized, Role: FOLLOWER
I20240503 01:59:05.942375 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } }
I20240503 01:59:05.942606 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:05.942732 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:05.942888 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:05.943572 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } }
I20240503 01:59:05.943913 477 leader_election.cc:302] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0045eb859f304fbd9ffc8c7eda824b04; no voters:
I20240503 01:59:05.944182 477 leader_election.cc:288] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:05.944298 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:05.945025 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [term 1 LEADER]: Becoming Leader. State: Replica: 0045eb859f304fbd9ffc8c7eda824b04, State: Running, Role: LEADER
I20240503 01:59:05.945335 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } }
I20240503 01:59:05.945387 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:05.947769 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0045eb859f304fbd9ffc8c7eda824b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } } }
I20240503 01:59:05.948042 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0045eb859f304fbd9ffc8c7eda824b04. Latest consensus state: current_term: 1 leader_uuid: "0045eb859f304fbd9ffc8c7eda824b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045eb859f304fbd9ffc8c7eda824b04" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35171 } } }
I20240503 01:59:05.948143 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:05.948854 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04 [sys.catalog]: This master's current role is: LEADER
W20240503 01:59:05.950229 496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:59:05.950294 496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:59:05.951818 498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:05.952508 498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:05.956950 498 catalog_manager.cc:1347] Generated new cluster ID: 3fb71986f6a148e6ae6a1d237dce9968
I20240503 01:59:05.957049 498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:05.968071 498 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:05.969118 498 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:05.975157 498 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0045eb859f304fbd9ffc8c7eda824b04: Generated new TSK 0
I20240503 01:59:05.975812 498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:05.979149 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:06.093353 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.093550 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.093621 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.096586 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.096652 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.096750 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:06.100091 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.100855 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.101888 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.110211 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.110903 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.111614 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.112430 501 server_base.cc:1034] running on GCE node
I20240503 01:59:06.112753 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.113273 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.114416 501 hybrid_clock.cc:648] HybridClock initialized: now 1714701546114400 us; error 35 us; skew 500 ppm
I20240503 01:59:06.116283 501 webserver.cc:460] Webserver started at http://127.0.99.129:35049/ using document root <none> and password file <none>
I20240503 01:59:06.116854 501 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.116911 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.117086 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.118647 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0a141082e3c24126a1542e2c9a3aad0e"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.119105 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0a141082e3c24126a1542e2c9a3aad0e"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.122305 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:06.124462 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.125464 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:06.125562 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0a141082e3c24126a1542e2c9a3aad0e"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.125638 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.141685 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.142200 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.142347 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.142824 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.143659 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.143707 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.143780 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.143815 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.152328 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39079
I20240503 01:59:06.152374 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39079 every 8 connection(s)
I20240503 01:59:06.153273 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:06.154801 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 501
I20240503 01:59:06.154914 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:06.157675 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:06.163043 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:06.163290 629 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:06.164039 629 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:06.165539 441 ts_manager.cc:194] Registered new tserver with Master: 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:06.166707 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:48383
W20240503 01:59:06.262447 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.262645 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.262691 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.265661 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.265715 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.265810 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:06.269237 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.270046 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.271061 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.278843 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.279625 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.280388 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.281082 632 server_base.cc:1034] running on GCE node
I20240503 01:59:06.281414 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.281911 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.283077 632 hybrid_clock.cc:648] HybridClock initialized: now 1714701546283064 us; error 29 us; skew 500 ppm
I20240503 01:59:06.284850 632 webserver.cc:460] Webserver started at http://127.0.99.130:37515/ using document root <none> and password file <none>
I20240503 01:59:06.285375 632 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.285434 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.285609 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.287338 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "efd7ce56deab4305a002577febb80532"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.287902 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "efd7ce56deab4305a002577febb80532"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.291301 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:06.293519 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.294512 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:06.294613 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "efd7ce56deab4305a002577febb80532"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.294696 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.308357 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.308862 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.309002 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.309450 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.310248 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.310294 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.310346 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.310377 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.319684 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46503
I20240503 01:59:06.319710 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46503 every 8 connection(s)
I20240503 01:59:06.320564 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:06.323035 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 632
I20240503 01:59:06.323143 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:06.325459 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:06.330405 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:06.330610 760 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:06.331099 760 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:06.332130 441 ts_manager.cc:194] Registered new tserver with Master: efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:06.332916 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43893
W20240503 01:59:06.430141 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.430333 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.430377 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.433310 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.433367 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.433435 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:06.436828 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.437556 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.438566 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.446399 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.447062 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.447988 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.448563 763 server_base.cc:1034] running on GCE node
I20240503 01:59:06.448904 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.449400 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.450558 763 hybrid_clock.cc:648] HybridClock initialized: now 1714701546450549 us; error 21 us; skew 500 ppm
I20240503 01:59:06.452345 763 webserver.cc:460] Webserver started at http://127.0.99.131:38615/ using document root <none> and password file <none>
I20240503 01:59:06.452845 763 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.452903 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.453063 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.454560 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c0801789471740eda4246b1ab4904e10"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.455001 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c0801789471740eda4246b1ab4904e10"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.458113 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 01:59:06.460165 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.461069 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:06.461171 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c0801789471740eda4246b1ab4904e10"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.461248 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.482254 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.482765 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.482904 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.483393 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.484724 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.484771 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.484824 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.484853 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.493472 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38797
I20240503 01:59:06.493518 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38797 every 8 connection(s)
I20240503 01:59:06.494330 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:06.500985 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 763
I20240503 01:59:06.501081 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:06.503101 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:06.503258 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:06.503304 891 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:06.503855 891 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:06.504741 441 ts_manager.cc:194] Registered new tserver with Master: c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:06.505275 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:37551
W20240503 01:59:06.611773 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.611975 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.612028 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.614926 894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.614980 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.615085 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:06.618503 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.619258 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.620280 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.628027 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.628916 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.629657 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.630398 894 server_base.cc:1034] running on GCE node
I20240503 01:59:06.630756 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.631239 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.632385 894 hybrid_clock.cc:648] HybridClock initialized: now 1714701546632375 us; error 25 us; skew 500 ppm
I20240503 01:59:06.634061 894 webserver.cc:460] Webserver started at http://127.0.99.132:39229/ using document root <none> and password file <none>
I20240503 01:59:06.634562 894 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.634619 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.634780 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.636332 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "648ac6ef81784cf3a916a2ed46655b3f"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.636767 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "648ac6ef81784cf3a916a2ed46655b3f"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.639714 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:06.641712 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.642623 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:06.642717 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "648ac6ef81784cf3a916a2ed46655b3f"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.642792 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.656122 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.656564 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.656706 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.657131 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.657888 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.657939 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.657994 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.658021 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.666568 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34053
I20240503 01:59:06.666615 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34053 every 8 connection(s)
I20240503 01:59:06.667421 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:06.668334 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 894
I20240503 01:59:06.668442 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:06.670361 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:06.676268 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:06.676501 1022 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:06.677122 1022 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:06.678123 441 ts_manager.cc:194] Registered new tserver with Master: 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:06.678791 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:48553
W20240503 01:59:06.778246 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.778436 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.778479 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.781383 1025 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.781436 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.781500 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:06.785180 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.785941 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.786940 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.795892 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.796712 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.797123 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.797925 1025 server_base.cc:1034] running on GCE node
I20240503 01:59:06.798334 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.798883 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.800029 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714701546800020 us; error 23 us; skew 500 ppm
I20240503 01:59:06.801846 1025 webserver.cc:460] Webserver started at http://127.0.99.133:33787/ using document root <none> and password file <none>
I20240503 01:59:06.802397 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.802454 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.802625 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.804179 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "091deff9eeda46b4ab93b05a3f466847"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.804613 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "091deff9eeda46b4ab93b05a3f466847"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.807662 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:06.809653 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.810564 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:06.810658 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "091deff9eeda46b4ab93b05a3f466847"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.810736 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.824553 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.825119 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.825292 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.825784 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.826651 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.826704 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.826766 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.826797 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.836876 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36505
I20240503 01:59:06.836942 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36505 every 8 connection(s)
I20240503 01:59:06.837828 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:06.845633 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:06.845731 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 1025
I20240503 01:59:06.845839 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:06.845834 1153 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:06.846378 1153 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:06.847339 440 ts_manager.cc:194] Registered new tserver with Master: 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505)
I20240503 01:59:06.847930 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:54889
I20240503 01:59:06.848148 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:06.955225 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:06.955415 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:06.955457 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:06.958395 1156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:06.958454 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:06.958519 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:06.961943 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:06.962694 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:06.963657 1156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:06.971933 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.972422 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:06.973122 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:06.973868 1156 server_base.cc:1034] running on GCE node
I20240503 01:59:06.974236 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:06.974730 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:06.975881 1156 hybrid_clock.cc:648] HybridClock initialized: now 1714701546975857 us; error 41 us; skew 500 ppm
I20240503 01:59:06.977630 1156 webserver.cc:460] Webserver started at http://127.0.99.134:36537/ using document root <none> and password file <none>
I20240503 01:59:06.978144 1156 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:06.978201 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:06.978359 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:06.979935 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e139f2784fa5493983f786b885d7c33d"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.980373 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e139f2784fa5493983f786b885d7c33d"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.983382 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:06.985343 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.986250 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:06.986346 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e139f2784fa5493983f786b885d7c33d"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:06.986419 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:06.997081 1156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:06.997534 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:06.997678 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:06.998149 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:06.998942 1156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:06.998996 1156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:06.999054 1156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:06.999084 1156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.007839 1156 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34873
I20240503 01:59:07.007927 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34873 every 8 connection(s)
I20240503 01:59:07.008654 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:07.013145 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 1156
I20240503 01:59:07.013252 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:07.015676 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:07.017655 1284 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:07.017896 1284 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:07.018476 1284 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:07.019403 440 ts_manager.cc:194] Registered new tserver with Master: e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873)
I20240503 01:59:07.019951 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50629
W20240503 01:59:07.120369 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:07.120561 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:07.120635 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:07.123492 1287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:07.123585 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:07.123708 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:07.127097 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:07.127925 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:07.128969 1287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:07.137678 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:07.138396 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:07.139142 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:07.139832 1287 server_base.cc:1034] running on GCE node
I20240503 01:59:07.140203 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:07.140729 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:07.141891 1287 hybrid_clock.cc:648] HybridClock initialized: now 1714701547141880 us; error 23 us; skew 500 ppm
I20240503 01:59:07.143563 1287 webserver.cc:460] Webserver started at http://127.0.99.135:41157/ using document root <none> and password file <none>
I20240503 01:59:07.144173 1287 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:07.144229 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:07.144400 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:07.146010 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "07d11a6a41ef47b888dae403ffbf6456"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.146512 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "07d11a6a41ef47b888dae403ffbf6456"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.149677 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 01:59:07.151655 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.152716 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:07.152813 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "07d11a6a41ef47b888dae403ffbf6456"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.152892 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:07.169175 629 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:07.170722 1287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:07.171240 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:07.171384 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:07.171914 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:07.172724 1287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:07.172770 1287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.172832 1287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:07.172857 1287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.182029 1287 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37161
I20240503 01:59:07.182081 1414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37161 every 8 connection(s)
I20240503 01:59:07.182901 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:07.191016 1415 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:07.191200 1415 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:07.191637 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 1287
I20240503 01:59:07.191648 1415 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:07.191717 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:07.192606 440 ts_manager.cc:194] Registered new tserver with Master: 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:07.193174 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:48585
I20240503 01:59:07.193657 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35171
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:07.301740 1418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:07.301934 1418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:07.301977 1418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:07.304898 1418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:07.304956 1418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:07.305050 1418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:07.308598 1418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.1418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:07.309419 1418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:07.310429 1418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:07.318437 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:07.319149 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:07.319932 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:07.320672 1418 server_base.cc:1034] running on GCE node
I20240503 01:59:07.321035 1418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:07.321535 1418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:07.322695 1418 hybrid_clock.cc:648] HybridClock initialized: now 1714701547322677 us; error 37 us; skew 500 ppm
I20240503 01:59:07.324443 1418 webserver.cc:460] Webserver started at http://127.0.99.136:37587/ using document root <none> and password file <none>
I20240503 01:59:07.324949 1418 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:07.325002 1418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:07.325168 1418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:07.326720 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b6db6ee3749340658049b2301a938993"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.327173 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b6db6ee3749340658049b2301a938993"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.330355 1418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:07.332343 1432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.333259 1418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:07.333348 1418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b6db6ee3749340658049b2301a938993"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:07.333432 1418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:07.334653 760 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:07.350448 1418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:07.350942 1418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:07.351087 1418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:07.351540 1418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:07.352344 1418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:07.352391 1418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.352447 1418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:07.352478 1418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:07.361302 1418 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34021
I20240503 01:59:07.361325 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34021 every 8 connection(s)
I20240503 01:59:07.362140 1418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:07.369087 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 1418
I20240503 01:59:07.369208 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:07.370916 1546 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:07.371124 1546 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:07.371711 1546 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:07.372709 440 ts_manager.cc:194] Registered new tserver with Master: b6db6ee3749340658049b2301a938993 (127.0.99.136:34021)
I20240503 01:59:07.373208 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:54705
I20240503 01:59:07.383576 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:07.394119 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:07.394203 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 1025
I20240503 01:59:07.399605 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 1156
I20240503 01:59:07.407369 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 1287
I20240503 01:59:07.412878 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 1418
I20240503 01:59:07.506978 891 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:07.681084 1022 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:11.170217 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51382:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:11.190336 694 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb1c96b35a4b98a1edfa4ccd99602d (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.190336 825 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb1c96b35a4b98a1edfa4ccd99602d (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.191172 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.191200 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.192028 824 tablet_service.cc:1467] Processing CreateTablet for tablet e6835fae0d5049f1a7e31fa51fd58f7a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.191830 563 tablet_service.cc:1467] Processing CreateTablet for tablet 42cb1c96b35a4b98a1edfa4ccd99602d (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.192268 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835fae0d5049f1a7e31fa51fd58f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.192607 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.196038 562 tablet_service.cc:1467] Processing CreateTablet for tablet 66c65746b14f430ca238ee3a9f5cf008 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.196033 693 tablet_service.cc:1467] Processing CreateTablet for tablet 66c65746b14f430ca238ee3a9f5cf008 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.196066 561 tablet_service.cc:1467] Processing CreateTablet for tablet e6835fae0d5049f1a7e31fa51fd58f7a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.196302 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c65746b14f430ca238ee3a9f5cf008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.196312 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c65746b14f430ca238ee3a9f5cf008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.200070 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835fae0d5049f1a7e31fa51fd58f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.200592 1568 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:11.202543 1568 tablet_bootstrap.cc:654] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.203186 1568 log.cc:824] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.203428 955 tablet_service.cc:1467] Processing CreateTablet for tablet e6835fae0d5049f1a7e31fa51fd58f7a (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.203428 956 tablet_service.cc:1467] Processing CreateTablet for tablet 66c65746b14f430ca238ee3a9f5cf008 (DEFAULT_TABLE table=rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.204026 1567 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:11.204386 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835fae0d5049f1a7e31fa51fd58f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.204720 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c65746b14f430ca238ee3a9f5cf008. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.205909 1567 tablet_bootstrap.cc:654] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.206641 1567 log.cc:824] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.207299 1566 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:11.208937 1568 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:11.208935 1567 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:11.209106 1567 ts_tablet_manager.cc:1367] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:11.209141 1568 ts_tablet_manager.cc:1367] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 01:59:11.209228 1566 tablet_bootstrap.cc:654] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.209919 1566 log.cc:824] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.210547 1571 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:11.212074 1571 tablet_bootstrap.cc:654] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.212749 1571 log.cc:824] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:11.213147 1566 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:11.213325 1566 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:11.213361 1568 raft_consensus.cc:348] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.213699 1568 raft_consensus.cc:374] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.213793 1568 raft_consensus.cc:729] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.213692 1567 raft_consensus.cc:348] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.213886 1567 raft_consensus.cc:374] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.213932 1567 raft_consensus.cc:729] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.214370 1567 consensus_queue.cc:260] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.214371 1568 consensus_queue.cc:260] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.214987 1571 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:11.215134 1571 ts_tablet_manager.cc:1367] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:11.215216 1567 ts_tablet_manager.cc:1398] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:59:11.215323 1568 ts_tablet_manager.cc:1398] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:11.215612 1568 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:11.216405 1567 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:11.216593 1568 tablet_bootstrap.cc:654] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.217397 1567 tablet_bootstrap.cc:654] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.217850 1568 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:11.217978 1568 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:11.218385 1568 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.218472 1568 raft_consensus.cc:374] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.218514 1568 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.218660 1568 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.218640 1571 raft_consensus.cc:348] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.218839 1571 raft_consensus.cc:374] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.218894 1571 raft_consensus.cc:729] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.218933 1568 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.219313 1571 consensus_queue.cc:260] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.220124 1571 ts_tablet_manager.cc:1398] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 01:59:11.220141 1567 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:11.220292 1567 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:11.220417 1571 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:11.220872 1567 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.220958 1567 raft_consensus.cc:374] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.220989 1567 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.221132 1567 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.221369 1567 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.221392 1571 tablet_bootstrap.cc:654] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.222445 1571 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:11.222568 1571 ts_tablet_manager.cc:1367] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:11.223114 1571 raft_consensus.cc:348] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.223327 1571 raft_consensus.cc:374] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.223387 1571 raft_consensus.cc:729] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.223512 1571 consensus_queue.cc:260] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.223798 1571 ts_tablet_manager.cc:1398] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.224145 1566 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.224354 1566 raft_consensus.cc:374] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.224428 1566 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.224797 1566 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.225616 1566 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240503 01:59:11.225847 1566 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:11.226804 1566 tablet_bootstrap.cc:654] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.227840 1566 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:11.227952 1566 ts_tablet_manager.cc:1367] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:11.228403 1566 raft_consensus.cc:348] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.228482 1566 raft_consensus.cc:374] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.228502 1566 raft_consensus.cc:729] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.228597 1566 consensus_queue.cc:260] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.228833 1566 ts_tablet_manager.cc:1398] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:11.228969 1566 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:11.229784 1566 tablet_bootstrap.cc:654] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.230748 1566 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:11.230823 1566 ts_tablet_manager.cc:1367] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:11.231189 1566 raft_consensus.cc:348] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.231277 1566 raft_consensus.cc:374] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.231310 1566 raft_consensus.cc:729] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.231422 1566 consensus_queue.cc:260] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.231633 1566 ts_tablet_manager.cc:1398] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:11.235420 1574 raft_consensus.cc:482] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.235570 1574 raft_consensus.cc:504] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.236609 1574 leader_election.cc:288] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:11.241341 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:11.241343 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:11.241596 845 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 0.
I20240503 01:59:11.241596 583 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 0.
I20240503 01:59:11.242003 650 leader_election.cc:302] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.242213 1574 raft_consensus.cc:2781] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.242285 1574 raft_consensus.cc:482] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.242331 1574 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:11.248473 892 tablet.cc:2369] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:11.257278 1580 raft_consensus.cc:482] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.257383 1580 raft_consensus.cc:504] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.258359 1580 leader_election.cc:288] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.262610 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:11.262744 845 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:11.263056 519 leader_election.cc:302] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:11.263074 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:11.263304 1580 raft_consensus.cc:2781] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.263340 976 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:11.263372 1580 raft_consensus.cc:482] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.263404 1580 raft_consensus.cc:3037] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.267019 1574 raft_consensus.cc:504] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.267021 1580 raft_consensus.cc:504] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.267444 1580 leader_election.cc:288] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.267447 1574 leader_election.cc:288] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:11.267992 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:11.268002 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:11.267985 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:11.268100 583 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.268105 976 raft_consensus.cc:3037] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.268110 845 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.269102 976 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:11.269102 583 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:11.269354 845 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:11.269421 1578 raft_consensus.cc:482] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.269497 649 leader_election.cc:302] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.269510 1578 raft_consensus.cc:504] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.269661 1574 raft_consensus.cc:2781] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.269866 1574 raft_consensus.cc:686] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Becoming Leader. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:11.270005 519 leader_election.cc:302] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:11.270206 1580 raft_consensus.cc:2781] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.270205 1574 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.270412 1580 raft_consensus.cc:686] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 1 LEADER]: Becoming Leader. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Running, Role: LEADER
I20240503 01:59:11.270627 1578 leader_election.cc:288] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:11.270710 1580 consensus_queue.cc:237] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:11.272607 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:11.272732 845 raft_consensus.cc:3037] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.273391 440 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 reported cstate change: term changed from 0 to 1, leader changed from <none> to efd7ce56deab4305a002577febb80532 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.273800 441 catalog_manager.cc:5484] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0a141082e3c24126a1542e2c9a3aad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:11.277942 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c65746b14f430ca238ee3a9f5cf008" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:11.278053 583 raft_consensus.cc:2454] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:11.278234 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c65746b14f430ca238ee3a9f5cf008" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:11.278347 911 leader_election.cc:302] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:11.278407 714 raft_consensus.cc:2454] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:11.278585 1578 raft_consensus.cc:2781] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.278666 1578 raft_consensus.cc:482] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.278734 1578 raft_consensus.cc:3037] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:11.325139 761 tablet.cc:2369] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:11.368052 845 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:11.368090 1578 raft_consensus.cc:504] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.368585 1578 leader_election.cc:288] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:11.369006 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c65746b14f430ca238ee3a9f5cf008" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:11.369073 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c65746b14f430ca238ee3a9f5cf008" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:11.369129 583 raft_consensus.cc:3037] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.369218 714 raft_consensus.cc:3037] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.370188 583 raft_consensus.cc:2454] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:11.370396 714 raft_consensus.cc:2454] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:11.370657 911 leader_election.cc:302] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:11.370932 1578 raft_consensus.cc:2781] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.371167 1578 raft_consensus.cc:686] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:11.371548 1578 consensus_queue.cc:237] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:11.375062 440 catalog_manager.cc:5484] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:11.408020 630 tablet.cc:2369] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:11.421751 1023 tablet.cc:2369] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:11.522711 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:11.522977 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:11.526000 1596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:11.616206 583 raft_consensus.cc:1261] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.616204 714 raft_consensus.cc:1261] T 66c65746b14f430ca238ee3a9f5cf008 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.617177 845 raft_consensus.cc:1261] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.618348 976 raft_consensus.cc:1261] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.618808 1594 consensus_queue.cc:1035] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.618825 1580 consensus_queue.cc:1035] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.619081 583 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.619392 845 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:11.619511 1587 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.619871 1578 consensus_queue.cc:1035] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.619885 1580 consensus_queue.cc:1035] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.623732 1574 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:11.635936 1628 mvcc.cc:205] Tried to move back new op lower bound from 7023417555414933504 to 7023417554005458944. Current Snapshot: MvccSnapshot[applied={T|T < 7023417555414933504}]
I20240503 01:59:11.636893 1629 mvcc.cc:205] Tried to move back new op lower bound from 7023417555432046592 to 7023417554007384064. Current Snapshot: MvccSnapshot[applied={T|T < 7023417555412643840}]
I20240503 01:59:11.637938 1637 mvcc.cc:205] Tried to move back new op lower bound from 7023417555412643840 to 7023417554007384064. Current Snapshot: MvccSnapshot[applied={T|T < 7023417555412643840}]
I20240503 01:59:11.640085 1627 mvcc.cc:205] Tried to move back new op lower bound from 7023417555427201024 to 7023417554421379072. Current Snapshot: MvccSnapshot[applied={T|T < 7023417555412799488}]
INSERT report
rows total: 2000
time total: 124.212 ms
time per row: 0.0621061 ms
I20240503 01:59:11.696830 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51382:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:11.702105 694 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd5de36fcad46a0bf6aca04394e46c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.702287 956 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd5de36fcad46a0bf6aca04394e46c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.702399 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd5de36fcad46a0bf6aca04394e46c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.702527 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd5de36fcad46a0bf6aca04394e46c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.702977 955 tablet_service.cc:1467] Processing CreateTablet for tablet a63bd458e2e94636a3a64ddc59ad8196 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.702996 824 tablet_service.cc:1467] Processing CreateTablet for tablet a63bd458e2e94636a3a64ddc59ad8196 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.703212 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63bd458e2e94636a3a64ddc59ad8196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.703217 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63bd458e2e94636a3a64ddc59ad8196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.704010 825 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd5de36fcad46a0bf6aca04394e46c7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.704232 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd5de36fcad46a0bf6aca04394e46c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.705125 1571 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:11.705284 823 tablet_service.cc:1467] Processing CreateTablet for tablet 8018cb49863d4736ae5f79f472f63bb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.705482 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8018cb49863d4736ae5f79f472f63bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.705998 1571 tablet_bootstrap.cc:654] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.706090 1568 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:11.706959 1568 tablet_bootstrap.cc:654] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.707072 1571 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:11.707145 1571 ts_tablet_manager.cc:1367] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:11.707348 1567 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:11.707588 1571 raft_consensus.cc:348] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.707617 561 tablet_service.cc:1467] Processing CreateTablet for tablet 8018cb49863d4736ae5f79f472f63bb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.707684 1571 raft_consensus.cc:374] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.707736 1571 raft_consensus.cc:729] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.707916 1571 consensus_queue.cc:260] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.707966 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8018cb49863d4736ae5f79f472f63bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.708230 1571 ts_tablet_manager.cc:1398] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:11.708245 1567 tablet_bootstrap.cc:654] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.708393 1571 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:11.709568 1571 tablet_bootstrap.cc:654] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.709828 1566 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:11.710433 1567 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:11.710517 1567 ts_tablet_manager.cc:1367] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:11.710633 1566 tablet_bootstrap.cc:654] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.711014 1571 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:11.710994 1567 raft_consensus.cc:348] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.711086 1571 ts_tablet_manager.cc:1367] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:11.711093 1567 raft_consensus.cc:374] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.711126 1567 raft_consensus.cc:729] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.711243 1567 consensus_queue.cc:260] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.711488 1567 ts_tablet_manager.cc:1398] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:11.711479 1571 raft_consensus.cc:348] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.711521 693 tablet_service.cc:1467] Processing CreateTablet for tablet 8018cb49863d4736ae5f79f472f63bb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.711575 1571 raft_consensus.cc:374] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.711613 1571 raft_consensus.cc:729] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.711716 1571 consensus_queue.cc:260] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.711791 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8018cb49863d4736ae5f79f472f63bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.711964 1571 ts_tablet_manager.cc:1398] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.713786 1566 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:11.713855 1567 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:11.713866 1566 ts_tablet_manager.cc:1367] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:11.714181 1566 raft_consensus.cc:348] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.714267 1566 raft_consensus.cc:374] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.714305 1566 raft_consensus.cc:729] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.714426 1566 consensus_queue.cc:260] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.714670 1566 ts_tablet_manager.cc:1398] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.714712 1567 tablet_bootstrap.cc:654] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.715029 694 tablet_service.cc:1467] Processing CreateTablet for tablet a63bd458e2e94636a3a64ddc59ad8196 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:11.715267 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63bd458e2e94636a3a64ddc59ad8196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:11.715813 1568 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:11.715899 1568 ts_tablet_manager.cc:1367] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:11.716336 1568 raft_consensus.cc:348] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.716430 1568 raft_consensus.cc:374] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.716457 1568 raft_consensus.cc:729] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.716579 1568 consensus_queue.cc:260] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.716859 1568 ts_tablet_manager.cc:1398] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:11.717021 1568 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:11.717948 1567 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:11.717983 1568 tablet_bootstrap.cc:654] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.718029 1567 ts_tablet_manager.cc:1367] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:11.718335 1567 raft_consensus.cc:348] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.718415 1567 raft_consensus.cc:374] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.718447 1567 raft_consensus.cc:729] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.718572 1567 consensus_queue.cc:260] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.718853 1567 ts_tablet_manager.cc:1398] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:11.718942 1568 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:11.719009 1568 ts_tablet_manager.cc:1367] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:11.719014 1567 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:11.719413 1568 raft_consensus.cc:348] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.719499 1568 raft_consensus.cc:374] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.719530 1568 raft_consensus.cc:729] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.719641 1568 consensus_queue.cc:260] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.719795 1567 tablet_bootstrap.cc:654] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.720132 1568 ts_tablet_manager.cc:1398] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.720268 1568 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:11.720772 1567 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:11.720847 1567 ts_tablet_manager.cc:1367] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:11.721112 1568 tablet_bootstrap.cc:654] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:11.721287 1567 raft_consensus.cc:348] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.721390 1567 raft_consensus.cc:374] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.721427 1567 raft_consensus.cc:729] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.721557 1567 consensus_queue.cc:260] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.721905 1567 ts_tablet_manager.cc:1398] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:11.722415 1568 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:11.722489 1568 ts_tablet_manager.cc:1367] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:11.722888 1568 raft_consensus.cc:348] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.722995 1568 raft_consensus.cc:374] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:11.723028 1568 raft_consensus.cc:729] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:11.723145 1568 consensus_queue.cc:260] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.723433 1568 ts_tablet_manager.cc:1398] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:11.739967 1575 raft_consensus.cc:482] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.740083 1575 raft_consensus.cc:504] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.741120 1575 leader_election.cc:288] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.745347 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd5de36fcad46a0bf6aca04394e46c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:11.745349 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd5de36fcad46a0bf6aca04394e46c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:11.745532 714 raft_consensus.cc:2454] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:11.745528 976 raft_consensus.cc:2454] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:11.745872 778 leader_election.cc:302] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.746155 1575 raft_consensus.cc:2781] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.746213 1575 raft_consensus.cc:482] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.746243 1575 raft_consensus.cc:3037] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.760694 1575 raft_consensus.cc:504] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.761180 1575 leader_election.cc:288] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Requested vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.761521 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd5de36fcad46a0bf6aca04394e46c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:11.761528 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd5de36fcad46a0bf6aca04394e46c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:11.761641 714 raft_consensus.cc:3037] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.761652 976 raft_consensus.cc:3037] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.762605 714 raft_consensus.cc:2454] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:11.762831 976 raft_consensus.cc:2454] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:11.762876 778 leader_election.cc:302] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.763026 1575 raft_consensus.cc:2781] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.763183 1575 raft_consensus.cc:686] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:11.763482 1575 consensus_queue.cc:237] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.766037 441 catalog_manager.cc:5484] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.820030 1587 raft_consensus.cc:482] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.820158 1587 raft_consensus.cc:504] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.820636 1587 leader_election.cc:288] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.821040 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63bd458e2e94636a3a64ddc59ad8196" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:11.821203 845 raft_consensus.cc:2454] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 0.
I20240503 01:59:11.821525 650 leader_election.cc:302] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.821693 1587 raft_consensus.cc:2781] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.821745 1587 raft_consensus.cc:482] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.821789 1587 raft_consensus.cc:3037] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.824133 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63bd458e2e94636a3a64ddc59ad8196" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:11.824249 976 raft_consensus.cc:2454] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 0.
I20240503 01:59:11.858387 1587 raft_consensus.cc:504] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.858799 1587 leader_election.cc:288] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:11.859210 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63bd458e2e94636a3a64ddc59ad8196" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:11.859302 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63bd458e2e94636a3a64ddc59ad8196" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:11.859347 845 raft_consensus.cc:3037] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.859426 976 raft_consensus.cc:3037] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.860363 845 raft_consensus.cc:2454] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:11.860363 976 raft_consensus.cc:2454] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:11.860671 650 leader_election.cc:302] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.860849 1587 raft_consensus.cc:2781] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.860970 1587 raft_consensus.cc:686] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Becoming Leader. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:11.861146 1587 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:11.862881 441 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 reported cstate change: term changed from 0 to 1, leader changed from <none> to efd7ce56deab4305a002577febb80532 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:11.886404 1575 raft_consensus.cc:482] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:11.886544 1575 raft_consensus.cc:504] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.887066 1575 leader_election.cc:288] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:11.887547 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8018cb49863d4736ae5f79f472f63bb8" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:11.887697 714 raft_consensus.cc:2454] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:11.887995 778 leader_election.cc:302] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:11.888176 1575 raft_consensus.cc:2781] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:11.888232 1575 raft_consensus.cc:482] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:11.888279 1575 raft_consensus.cc:3037] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.890528 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8018cb49863d4736ae5f79f472f63bb8" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:11.890653 583 raft_consensus.cc:2454] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:11.918213 1575 raft_consensus.cc:504] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.918625 1575 leader_election.cc:288] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:11.919067 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8018cb49863d4736ae5f79f472f63bb8" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:11.919077 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8018cb49863d4736ae5f79f472f63bb8" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:11.919184 583 raft_consensus.cc:3037] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.919215 714 raft_consensus.cc:3037] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:11.919988 583 raft_consensus.cc:2454] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:11.920277 780 leader_election.cc:302] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:11.920349 714 raft_consensus.cc:2454] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:11.920447 1575 raft_consensus.cc:2781] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:11.920526 1575 raft_consensus.cc:686] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:11.920670 1575 consensus_queue.cc:237] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:11.922281 441 catalog_manager.cc:5484] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:12.059657 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:12.059916 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:12.062867 1671 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:12.135583 976 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.136169 1587 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.141034 975 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.141762 583 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.142151 713 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.142340 714 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.142586 1702 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.142908 1575 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.142935 1666 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.143024 1702 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.143515 845 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:12.144348 1574 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 98.2109 ms
time per row: 0.0491054 ms
I20240503 01:59:12.204797 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51382:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:12.210397 694 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba19da3d1f4a4182fcc967a8ca6314 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.210723 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba19da3d1f4a4182fcc967a8ca6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.211120 561 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba19da3d1f4a4182fcc967a8ca6314 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.211184 562 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd70004ec0a4244ad89c59b0fab01c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.211376 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba19da3d1f4a4182fcc967a8ca6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.211701 955 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba19da3d1f4a4182fcc967a8ca6314 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.211807 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd70004ec0a4244ad89c59b0fab01c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.211983 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba19da3d1f4a4182fcc967a8ca6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.211980 563 tablet_service.cc:1467] Processing CreateTablet for tablet b0c46bbf4f1b435993ce1057b2b4de9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.212193 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.213435 1566 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:12.214130 823 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd70004ec0a4244ad89c59b0fab01c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.214222 956 tablet_service.cc:1467] Processing CreateTablet for tablet 3bd70004ec0a4244ad89c59b0fab01c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.214293 1566 tablet_bootstrap.cc:654] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.214365 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd70004ec0a4244ad89c59b0fab01c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.214448 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bd70004ec0a4244ad89c59b0fab01c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.215443 1566 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:12.215513 1566 ts_tablet_manager.cc:1367] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.211702 954 tablet_service.cc:1467] Processing CreateTablet for tablet b0c46bbf4f1b435993ce1057b2b4de9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.216030 1568 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:12.215986 1566 raft_consensus.cc:348] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.216082 1566 raft_consensus.cc:374] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.216125 1566 raft_consensus.cc:729] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.216248 1566 consensus_queue.cc:260] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.216534 1566 ts_tablet_manager.cc:1398] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.216681 1566 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:12.216703 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.216903 1568 tablet_bootstrap.cc:654] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.217502 1566 tablet_bootstrap.cc:654] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.217928 1568 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:12.218004 1568 ts_tablet_manager.cc:1367] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.218127 693 tablet_service.cc:1467] Processing CreateTablet for tablet b0c46bbf4f1b435993ce1057b2b4de9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.218331 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.218444 1568 raft_consensus.cc:348] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.218536 1568 raft_consensus.cc:374] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.218566 1568 raft_consensus.cc:729] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.218685 1568 consensus_queue.cc:260] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.218945 1568 ts_tablet_manager.cc:1398] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:12.220194 1567 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:12.221082 1567 tablet_bootstrap.cc:654] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.222038 1713 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:12.222697 1713 tablet_bootstrap.cc:654] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.223883 1567 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:12.223973 1567 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:59:12.224043 1566 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:12.224087 1713 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:12.224129 1566 ts_tablet_manager.cc:1367] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:12.224164 1713 ts_tablet_manager.cc:1367] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.224397 1567 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.224503 1567 raft_consensus.cc:374] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.224475 1566 raft_consensus.cc:348] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.224539 1567 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.224565 1713 raft_consensus.cc:348] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.224660 1713 raft_consensus.cc:374] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.224699 1713 raft_consensus.cc:729] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.224656 1567 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.224839 1713 consensus_queue.cc:260] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.224934 1567 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.225131 1567 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:12.225612 1713 ts_tablet_manager.cc:1398] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:12.225762 1713 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:12.226022 1567 tablet_bootstrap.cc:654] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.226552 1566 raft_consensus.cc:374] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.226616 1713 tablet_bootstrap.cc:654] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.226665 1566 raft_consensus.cc:729] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.226892 1566 consensus_queue.cc:260] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.227306 1566 ts_tablet_manager.cc:1398] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:12.227531 1566 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:12.227674 1713 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:12.227767 1713 ts_tablet_manager.cc:1367] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:12.228170 1713 raft_consensus.cc:348] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.228261 1713 raft_consensus.cc:374] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.228291 1713 raft_consensus.cc:729] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.228349 1567 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:12.228411 1567 ts_tablet_manager.cc:1367] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:12.228408 1713 consensus_queue.cc:260] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.228669 1713 ts_tablet_manager.cc:1398] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.228812 1713 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:12.228957 1567 raft_consensus.cc:348] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.229043 1566 tablet_bootstrap.cc:654] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.229041 1567 raft_consensus.cc:374] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.229125 1567 raft_consensus.cc:729] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.229315 1567 consensus_queue.cc:260] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.229683 1713 tablet_bootstrap.cc:654] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.230052 1566 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:12.230129 1566 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:12.230556 1566 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.230657 1566 raft_consensus.cc:374] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.230695 1566 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.230697 1567 ts_tablet_manager.cc:1398] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.230811 1566 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.230902 1713 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:12.230970 1713 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.231088 1566 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.231283 1713 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.231374 1713 raft_consensus.cc:374] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.231416 1713 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.231505 1713 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.231722 1713 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:12.263248 1702 raft_consensus.cc:482] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.263571 1702 raft_consensus.cc:504] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.264082 1702 leader_election.cc:288] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:12.264585 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd70004ec0a4244ad89c59b0fab01c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:12.264581 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd70004ec0a4244ad89c59b0fab01c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:12.264743 975 raft_consensus.cc:2454] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:12.264755 583 raft_consensus.cc:2454] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:12.265093 780 leader_election.cc:302] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.265357 1702 raft_consensus.cc:2781] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.265415 1702 raft_consensus.cc:482] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.265465 1702 raft_consensus.cc:3037] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.284561 1578 raft_consensus.cc:482] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.284684 1578 raft_consensus.cc:504] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.285090 1578 leader_election.cc:288] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.285481 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:12.285480 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:12.285622 583 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:12.285622 714 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:12.285888 911 leader_election.cc:302] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:12.286103 1578 raft_consensus.cc:2781] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.286159 1578 raft_consensus.cc:482] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.286195 1578 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.294741 1723 raft_consensus.cc:482] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.294822 1723 raft_consensus.cc:504] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.295154 1723 leader_election.cc:288] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.295502 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba19da3d1f4a4182fcc967a8ca6314" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:12.295521 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba19da3d1f4a4182fcc967a8ca6314" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:12.295640 583 raft_consensus.cc:2454] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:12.295637 714 raft_consensus.cc:2454] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 0.
I20240503 01:59:12.295924 909 leader_election.cc:302] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:12.296118 1723 raft_consensus.cc:2781] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.296175 1723 raft_consensus.cc:482] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.296201 1723 raft_consensus.cc:3037] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.345327 1702 raft_consensus.cc:504] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.345352 1578 raft_consensus.cc:504] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.345799 1578 leader_election.cc:288] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.345844 1702 leader_election.cc:288] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:12.346302 1723 raft_consensus.cc:504] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.346416 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:12.346522 714 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.346506 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:12.346635 583 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.346689 1723 leader_election.cc:288] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.347012 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd70004ec0a4244ad89c59b0fab01c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:12.347115 582 raft_consensus.cc:3037] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.347534 714 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:12.347811 583 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:12.348047 909 leader_election.cc:302] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:12.348069 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bd70004ec0a4244ad89c59b0fab01c7" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:12.348182 975 raft_consensus.cc:3037] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.348255 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba19da3d1f4a4182fcc967a8ca6314" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:12.348351 714 raft_consensus.cc:3037] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.348495 582 raft_consensus.cc:2454] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:12.348842 1723 raft_consensus.cc:2781] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:12.348990 1723 raft_consensus.cc:686] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:12.349165 714 raft_consensus.cc:2454] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:12.349150 1723 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.349239 975 raft_consensus.cc:2454] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:12.349431 909 leader_election.cc:302] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:12.349575 1578 raft_consensus.cc:2781] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:12.349586 781 leader_election.cc:302] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.349701 1578 raft_consensus.cc:686] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:12.349822 1578 consensus_queue.cc:237] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:12.350482 1702 raft_consensus.cc:2781] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:12.350495 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba19da3d1f4a4182fcc967a8ca6314" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:12.350595 1702 raft_consensus.cc:686] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:12.350610 582 raft_consensus.cc:3037] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.350744 1702 consensus_queue.cc:237] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.351936 433 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.352216 433 catalog_manager.cc:5484] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:12.352383 441 catalog_manager.cc:5484] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.422930 582 raft_consensus.cc:2454] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
W20240503 01:59:12.483199 1729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:12.483554 1729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:12.488255 1729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:12.572851 582 raft_consensus.cc:1261] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.572851 714 raft_consensus.cc:1261] T d7ba19da3d1f4a4182fcc967a8ca6314 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.572903 975 raft_consensus.cc:1261] T 3bd70004ec0a4244ad89c59b0fab01c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.573253 583 raft_consensus.cc:1261] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.573285 1578 consensus_queue.cc:1035] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.573645 1702 consensus_queue.cc:1035] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.573935 1703 consensus_queue.cc:1035] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.574508 1578 consensus_queue.cc:1035] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.588292 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.588778 1723 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:12.589708 581 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:12.590646 1724 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.731 ms
time per row: 0.0573655 ms
I20240503 01:59:12.655565 436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51382:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:12.660871 823 tablet_service.cc:1467] Processing CreateTablet for tablet 4f75e2ba8c494fc5bde411b10b687860 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.661098 694 tablet_service.cc:1467] Processing CreateTablet for tablet 4f75e2ba8c494fc5bde411b10b687860 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.661180 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f75e2ba8c494fc5bde411b10b687860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.661335 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f75e2ba8c494fc5bde411b10b687860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.661321 825 tablet_service.cc:1467] Processing CreateTablet for tablet bae517688d34428e9a4983829376002b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.661607 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae517688d34428e9a4983829376002b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.663276 1567 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:12.663399 824 tablet_service.cc:1467] Processing CreateTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.663436 954 tablet_service.cc:1467] Processing CreateTablet for tablet bae517688d34428e9a4983829376002b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.663607 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.663681 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae517688d34428e9a4983829376002b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.664297 1567 tablet_bootstrap.cc:654] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.664268 563 tablet_service.cc:1467] Processing CreateTablet for tablet 4f75e2ba8c494fc5bde411b10b687860 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.664549 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f75e2ba8c494fc5bde411b10b687860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.664798 562 tablet_service.cc:1467] Processing CreateTablet for tablet bae517688d34428e9a4983829376002b (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.664933 1568 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:12.665047 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae517688d34428e9a4983829376002b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.665906 956 tablet_service.cc:1467] Processing CreateTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.666124 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.666606 1566 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:12.667208 1713 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:12.667618 1566 tablet_bootstrap.cc:654] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.668576 694 tablet_service.cc:1467] Processing CreateTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:12.668804 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:12.691727 1713 tablet_bootstrap.cc:654] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.693135 1713 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:12.693239 1713 ts_tablet_manager.cc:1367] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 01:59:12.693744 1713 raft_consensus.cc:348] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.693851 1713 raft_consensus.cc:374] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.693886 1713 raft_consensus.cc:729] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.694007 1713 consensus_queue.cc:260] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.694283 1713 ts_tablet_manager.cc:1398] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.694429 1713 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:12.695377 1713 tablet_bootstrap.cc:654] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.697288 1713 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:12.697363 1713 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:12.697734 1713 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.697826 1713 raft_consensus.cc:374] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.697860 1713 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.698014 1713 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.698329 1713 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.713392 1568 tablet_bootstrap.cc:654] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.714596 1568 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:12.714697 1568 ts_tablet_manager.cc:1367] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240503 01:59:12.715173 1568 raft_consensus.cc:348] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.715303 1568 raft_consensus.cc:374] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.715348 1568 raft_consensus.cc:729] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.715469 1568 consensus_queue.cc:260] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.715723 1568 ts_tablet_manager.cc:1398] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:12.715948 1568 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:12.716316 1567 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:12.716398 1567 ts_tablet_manager.cc:1367] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.002s
I20240503 01:59:12.716751 1566 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:12.716792 1568 tablet_bootstrap.cc:654] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.716831 1566 ts_tablet_manager.cc:1367] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.001s
I20240503 01:59:12.717216 1566 raft_consensus.cc:348] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.717332 1566 raft_consensus.cc:374] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.717373 1566 raft_consensus.cc:729] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.717518 1566 consensus_queue.cc:260] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.717829 1566 ts_tablet_manager.cc:1398] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:12.717890 1568 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:12.717854 1567 raft_consensus.cc:348] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.717948 1567 raft_consensus.cc:374] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.717953 1568 ts_tablet_manager.cc:1367] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:12.717983 1567 raft_consensus.cc:729] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.718009 1566 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:12.718117 1567 consensus_queue.cc:260] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.718417 1567 ts_tablet_manager.cc:1398] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:12.718379 1568 raft_consensus.cc:348] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.718581 1567 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:12.718469 1568 raft_consensus.cc:374] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.718874 1568 raft_consensus.cc:729] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.718924 1566 tablet_bootstrap.cc:654] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.719002 1568 consensus_queue.cc:260] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.719233 1568 ts_tablet_manager.cc:1398] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.719374 1567 tablet_bootstrap.cc:654] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.719393 1568 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:12.720237 1568 tablet_bootstrap.cc:654] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:12.721570 1568 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:12.721653 1568 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:12.721833 1566 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:12.721907 1566 ts_tablet_manager.cc:1367] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:59:12.722093 1568 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.722193 1568 raft_consensus.cc:374] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.722225 1568 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.722326 1568 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.722335 1566 raft_consensus.cc:348] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.722422 1566 raft_consensus.cc:374] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.722450 1566 raft_consensus.cc:729] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.722554 1568 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:12.722575 1566 consensus_queue.cc:260] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.723246 1566 ts_tablet_manager.cc:1398] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:12.723965 1567 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:12.724049 1567 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:12.724442 1567 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.724547 1567 raft_consensus.cc:374] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:12.724596 1567 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:12.724732 1567 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.725009 1567 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:12.767567 1580 raft_consensus.cc:482] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.767729 1580 raft_consensus.cc:504] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.768312 1580 leader_election.cc:288] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.768767 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:12.768936 845 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:12.769282 519 leader_election.cc:302] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.769459 1580 raft_consensus.cc:2781] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.769522 1580 raft_consensus.cc:482] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.769577 1580 raft_consensus.cc:3037] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.772099 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:12.772221 714 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:12.776834 1580 raft_consensus.cc:504] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.777171 1580 leader_election.cc:288] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.777503 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:12.777611 714 raft_consensus.cc:3037] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.777576 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:12.777740 845 raft_consensus.cc:3037] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.778416 714 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:12.778679 516 leader_election.cc:302] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:12.778854 1580 raft_consensus.cc:2781] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:12.778901 845 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:12.779006 1580 raft_consensus.cc:686] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 LEADER]: Becoming Leader. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Running, Role: LEADER
I20240503 01:59:12.779155 1580 consensus_queue.cc:237] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.781172 441 catalog_manager.cc:5484] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0a141082e3c24126a1542e2c9a3aad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.861765 1702 raft_consensus.cc:482] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.861900 1702 raft_consensus.cc:504] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.862288 1702 leader_election.cc:288] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.862659 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:12.862692 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:12.862795 975 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:12.862829 714 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:12.863106 781 leader_election.cc:302] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.863291 1702 raft_consensus.cc:2781] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.863348 1702 raft_consensus.cc:482] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.863397 1702 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.932950 1702 raft_consensus.cc:504] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.933528 1702 leader_election.cc:288] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:12.933905 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:12.934036 975 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.934052 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:12.934186 714 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:12.935004 975 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:12.935333 714 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:12.935459 781 leader_election.cc:302] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.935681 1702 raft_consensus.cc:2781] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:12.935775 1702 raft_consensus.cc:686] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:12.935906 1702 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:12.937681 441 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:12.962281 1778 raft_consensus.cc:482] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:12.962421 1778 raft_consensus.cc:504] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:12.962884 1778 leader_election.cc:288] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:12.963302 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:12.963407 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:12.963474 845 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:12.963578 975 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:12.963810 519 leader_election.cc:302] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:12.964008 1778 raft_consensus.cc:2781] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:12.964068 1778 raft_consensus.cc:482] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:12.964099 1778 raft_consensus.cc:3037] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.006878 1778 raft_consensus.cc:504] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.007491 1778 leader_election.cc:288] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:13.008000 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:13.008082 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:13.008136 845 raft_consensus.cc:3037] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.008229 975 raft_consensus.cc:3037] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.009037 845 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.009380 519 leader_election.cc:302] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:13.009387 975 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.009577 1778 raft_consensus.cc:2781] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:13.009712 1778 raft_consensus.cc:686] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 1 LEADER]: Becoming Leader. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Running, Role: LEADER
I20240503 01:59:13.009874 1778 consensus_queue.cc:237] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.011907 441 catalog_manager.cc:5484] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0a141082e3c24126a1542e2c9a3aad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: HEALTHY } } }
W20240503 01:59:13.145838 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:13.146140 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:13.149551 1782 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:13.250276 975 raft_consensus.cc:1261] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.250607 845 raft_consensus.cc:1261] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.250993 1778 consensus_queue.cc:1035] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.251134 1580 consensus_queue.cc:1035] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.252835 844 raft_consensus.cc:1261] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.252844 714 raft_consensus.cc:1261] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.253290 1580 consensus_queue.cc:1035] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.253496 1778 consensus_queue.cc:1035] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.259490 713 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.259969 1702 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.261044 975 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.262144 1702 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.514 ms
time per row: 0.0602571 ms
I20240503 01:59:13.319891 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51382:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:13.325551 694 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b3e448315456789bfa5953e80c7ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.325863 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b3e448315456789bfa5953e80c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.326404 824 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b3e448315456789bfa5953e80c7ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.326717 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b3e448315456789bfa5953e80c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.327037 562 tablet_service.cc:1467] Processing CreateTablet for tablet ba2b3e448315456789bfa5953e80c7ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.327210 563 tablet_service.cc:1467] Processing CreateTablet for tablet 481dd137b628482b9f1333aceb89ae39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.327302 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b3e448315456789bfa5953e80c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.327450 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481dd137b628482b9f1333aceb89ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.328812 694 tablet_service.cc:1467] Processing CreateTablet for tablet 481dd137b628482b9f1333aceb89ae39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.329043 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481dd137b628482b9f1333aceb89ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.329483 956 tablet_service.cc:1467] Processing CreateTablet for tablet 481dd137b628482b9f1333aceb89ae39 (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.329775 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481dd137b628482b9f1333aceb89ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.329881 1843 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:13.330773 1843 tablet_bootstrap.cc:654] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.331576 1844 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:13.331996 561 tablet_service.cc:1467] Processing CreateTablet for tablet 83a04a16e66b463b98976c490155139c (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.332175 1843 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:13.332259 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.332266 1843 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:13.332476 1844 tablet_bootstrap.cc:654] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.332773 1843 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.332873 1843 raft_consensus.cc:374] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.332906 1843 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.333034 1843 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.333518 1844 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:13.333606 1844 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:13.333914 1843 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:13.334036 1844 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.334120 1844 raft_consensus.cc:374] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.334141 1844 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.334245 1844 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.334499 1844 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:13.334609 1844 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:13.334684 954 tablet_service.cc:1467] Processing CreateTablet for tablet 83a04a16e66b463b98976c490155139c (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.334944 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.335242 1844 tablet_bootstrap.cc:654] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.337577 824 tablet_service.cc:1467] Processing CreateTablet for tablet 83a04a16e66b463b98976c490155139c (DEFAULT_TABLE table=rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:13.337538 1848 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:13.338022 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.338613 1848 tablet_bootstrap.cc:654] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.339685 1843 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Bootstrap starting.
I20240503 01:59:13.340166 1848 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:13.340333 1848 ts_tablet_manager.cc:1367] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:13.340734 1848 raft_consensus.cc:348] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.340847 1848 raft_consensus.cc:374] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.340859 1843 tablet_bootstrap.cc:654] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.340889 1848 raft_consensus.cc:729] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.341095 1848 consensus_queue.cc:260] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.341490 1848 ts_tablet_manager.cc:1398] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:13.341838 1848 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Bootstrap starting.
I20240503 01:59:13.342783 1848 tablet_bootstrap.cc:654] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.344717 1848 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: No bootstrap required, opened a new log
I20240503 01:59:13.344802 1848 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:13.345216 1848 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.345332 1848 raft_consensus.cc:374] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.345371 1848 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.345491 1848 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.345760 1848 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:13.347968 1842 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:13.348527 1844 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:13.348621 1844 ts_tablet_manager.cc:1367] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 01:59:13.348721 1843 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: No bootstrap required, opened a new log
I20240503 01:59:13.348806 1843 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:13.348871 1842 tablet_bootstrap.cc:654] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.349097 1844 raft_consensus.cc:348] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.349198 1844 raft_consensus.cc:374] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.349231 1844 raft_consensus.cc:729] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.349195 1843 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.349288 1843 raft_consensus.cc:374] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.349319 1843 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.349351 1844 consensus_queue.cc:260] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.349431 1843 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.349625 1844 ts_tablet_manager.cc:1398] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:13.349782 1844 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 01:59:13.350114 1843 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:13.350633 1844 tablet_bootstrap.cc:654] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.351828 1844 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: No bootstrap required, opened a new log
I20240503 01:59:13.351912 1844 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:13.352319 1844 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.352428 1844 raft_consensus.cc:374] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.352458 1844 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.352682 1778 raft_consensus.cc:482] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:13.353289 1778 raft_consensus.cc:504] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.353783 1778 leader_election.cc:288] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:13.354105 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:13.354220 975 raft_consensus.cc:2454] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:13.352571 1844 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.354583 519 leader_election.cc:302] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:13.354591 1844 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 01:59:13.354671 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:13.354929 1778 raft_consensus.cc:2781] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:13.354979 1778 raft_consensus.cc:482] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:13.355031 1778 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:13.356014 516 leader_election.cc:341] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:13.358938 1842 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:13.359026 1842 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 01:59:13.359391 1842 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.359486 1842 raft_consensus.cc:374] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.359519 1842 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.359655 1842 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.360021 1842 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:13.360177 1842 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532: Bootstrap starting.
I20240503 01:59:13.361030 1842 tablet_bootstrap.cc:654] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:13.362154 1842 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532: No bootstrap required, opened a new log
I20240503 01:59:13.362239 1842 ts_tablet_manager.cc:1367] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:13.362702 1842 raft_consensus.cc:348] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.362811 1842 raft_consensus.cc:374] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:13.362877 1842 raft_consensus.cc:729] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.363044 1842 consensus_queue.cc:260] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.363291 1842 ts_tablet_manager.cc:1398] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:13.367599 1778 raft_consensus.cc:504] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.368115 1778 leader_election.cc:288] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:13.368486 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:13.368618 975 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.369771 975 raft_consensus.cc:2454] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.370133 519 leader_election.cc:302] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f; no voters:
I20240503 01:59:13.370379 1778 raft_consensus.cc:2781] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:13.370617 1778 raft_consensus.cc:686] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 LEADER]: Becoming Leader. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Running, Role: LEADER
I20240503 01:59:13.370854 1778 consensus_queue.cc:237] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.372824 436 catalog_manager.cc:5484] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0a141082e3c24126a1542e2c9a3aad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:13.375923 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:13.376051 714 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.376880 714 raft_consensus.cc:2454] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.468101 1666 raft_consensus.cc:482] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:13.468233 1666 raft_consensus.cc:504] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.468652 1666 leader_election.cc:288] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:13.469125 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e" is_pre_election: true
I20240503 01:59:13.469141 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:13.469269 581 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:13.469293 975 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 0.
I20240503 01:59:13.469602 781 leader_election.cc:302] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:13.469789 1666 raft_consensus.cc:2781] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:13.469852 1666 raft_consensus.cc:482] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:13.469905 1666 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.547307 1666 raft_consensus.cc:504] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.547916 1666 leader_election.cc:288] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:13.548375 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:13.548403 581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a141082e3c24126a1542e2c9a3aad0e"
I20240503 01:59:13.548537 975 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.548537 581 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.549425 581 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:13.549708 780 leader_election.cc:302] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:13.549716 975 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:13.549899 1666 raft_consensus.cc:2781] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:13.549997 1666 raft_consensus.cc:686] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:13.550163 1666 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:13.552074 436 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:13.639577 1778 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:13.639720 1778 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.640272 1778 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:13.640683 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:13.640684 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:13.640818 714 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:13.640827 844 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 0.
I20240503 01:59:13.641120 519 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:13.641291 1778 raft_consensus.cc:2781] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:13.641371 1778 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:13.641427 1778 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.648109 1778 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.648459 1778 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Requested vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:13.648821 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:13.648825 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "0a141082e3c24126a1542e2c9a3aad0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:13.648943 844 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.648948 714 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:13.649821 844 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.650111 714 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a141082e3c24126a1542e2c9a3aad0e in term 1.
I20240503 01:59:13.650149 519 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:13.650303 1778 raft_consensus.cc:2781] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:13.650398 1778 raft_consensus.cc:686] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 1 LEADER]: Becoming Leader. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Running, Role: LEADER
I20240503 01:59:13.650534 1778 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:13.652385 436 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129). New cstate: current_term: 1 leader_uuid: "0a141082e3c24126a1542e2c9a3aad0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:13.812417 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:13.812667 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:13.815930 1858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:13.863900 1778 consensus_queue.cc:1035] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.869782 1830 consensus_queue.cc:1035] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.902602 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.903136 1778 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.905277 714 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 0a141082e3c24126a1542e2c9a3aad0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.910419 1778 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.912216 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.912590 1666 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.915441 581 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:13.916455 1666 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:13.938558 1894 mvcc.cc:205] Tried to move back new op lower bound from 7023417564831379456 to 7023417563342782464. Current Snapshot: MvccSnapshot[applied={T|T < 7023417564815941632}]
INSERT report
rows total: 2000
time total: 133.539 ms
time per row: 0.0667694 ms
I20240503 01:59:13.992583 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:36505
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:35171
--webserver_port=33787
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 01:59:14.102952 1918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:14.103155 1918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:14.103199 1918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:14.106493 1918 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:14.106549 1918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:14.106631 1918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:14.110340 1918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:36505
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=33787
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.1918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:14.111151 1918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:14.112249 1918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:14.121371 1923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.122025 1924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:14.122828 1918 server_base.cc:1034] running on GCE node
W20240503 01:59:14.123435 1926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:14.123837 1918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:14.124413 1918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:14.125567 1918 hybrid_clock.cc:648] HybridClock initialized: now 1714701554125533 us; error 47 us; skew 500 ppm
I20240503 01:59:14.127241 1918 webserver.cc:460] Webserver started at http://127.0.99.133:33787/ using document root <none> and password file <none>
I20240503 01:59:14.127799 1918 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:14.127875 1918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:14.131117 1918 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:14.132902 1932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.133858 1918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:14.133957 1918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "091deff9eeda46b4ab93b05a3f466847"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:14.134331 1918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:14.164665 1918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:14.165264 1918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:14.165421 1918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:14.166042 1918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:14.166975 1918 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:14.167033 1918 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.167098 1918 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:14.167129 1918 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.192371 1918 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36505
I20240503 01:59:14.192459 2045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36505 every 8 connection(s)
I20240503 01:59:14.193743 1918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:14.198803 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 1918
I20240503 01:59:14.199227 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:34873
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:35171
--webserver_port=36537
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:14.202809 2046 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:14.203018 2046 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:14.203522 2046 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:14.204622 436 ts_manager.cc:194] Re-registered known tserver with Master: 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505)
I20240503 01:59:14.205204 436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50671
W20240503 01:59:14.311295 2049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:14.311507 2049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:14.311576 2049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:14.314507 2049 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:14.314574 2049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:14.314666 2049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:14.318234 2049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:34873
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=36537
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.2049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:14.319057 2049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:14.320202 2049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:14.328787 2054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.329528 2057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.330425 2055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:14.331235 2049 server_base.cc:1034] running on GCE node
I20240503 01:59:14.331557 2049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:14.332207 2049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:14.333372 2049 hybrid_clock.cc:648] HybridClock initialized: now 1714701554333356 us; error 31 us; skew 500 ppm
I20240503 01:59:14.335191 2049 webserver.cc:460] Webserver started at http://127.0.99.134:36537/ using document root <none> and password file <none>
I20240503 01:59:14.335696 2049 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:14.335795 2049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:14.338999 2049 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:14.340770 2063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.341727 2049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:14.341820 2049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e139f2784fa5493983f786b885d7c33d"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 01:59:14.342212 2049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:14.366160 2049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:14.366784 2049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:14.366943 2049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:14.367465 2049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:14.368372 2049 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:14.368417 2049 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.368463 2049 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:14.368494 2049 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.377344 2049 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34873
I20240503 01:59:14.377362 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34873 every 8 connection(s)
I20240503 01:59:14.378307 2049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:14.384644 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 2049
I20240503 01:59:14.385016 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:37161
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:35171
--webserver_port=41157
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:14.388125 2177 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:14.388353 2177 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:14.388935 2177 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:14.390055 436 ts_manager.cc:194] Re-registered known tserver with Master: e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873)
I20240503 01:59:14.390851 436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:41669
W20240503 01:59:14.493261 2180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:14.493463 2180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:14.493532 2180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:14.496559 2180 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:14.496628 2180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:14.496714 2180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:14.500324 2180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:37161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41157
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.2180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:14.501158 2180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:14.502224 2180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:14.511358 2185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.512024 2186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.512800 2188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:14.513561 2180 server_base.cc:1034] running on GCE node
I20240503 01:59:14.513963 2180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:14.514554 2180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:14.515693 2180 hybrid_clock.cc:648] HybridClock initialized: now 1714701554515670 us; error 36 us; skew 500 ppm
I20240503 01:59:14.517747 2180 webserver.cc:460] Webserver started at http://127.0.99.135:41157/ using document root <none> and password file <none>
I20240503 01:59:14.518326 2180 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:14.518394 2180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:14.521620 2180 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:14.523448 2194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.524570 2180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:14.524662 2180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "07d11a6a41ef47b888dae403ffbf6456"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:14.525074 2180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:14.536063 2180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:14.536586 2180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:14.536736 2180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:14.537230 2180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:14.538098 2180 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:14.538142 2180 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.538203 2180 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:14.538234 2180 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.547245 2180 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37161
I20240503 01:59:14.547299 2307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37161 every 8 connection(s)
I20240503 01:59:14.548362 2180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:14.550629 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 2180
I20240503 01:59:14.551059 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:34021
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:35171
--webserver_port=37587
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 01:59:14.557873 2308 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:14.558115 2308 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:14.558715 2308 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:14.559804 436 ts_manager.cc:194] Re-registered known tserver with Master: 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:14.560544 436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:54631
W20240503 01:59:14.664083 2311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:14.664294 2311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:14.664359 2311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:14.667438 2311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 01:59:14.667507 2311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:14.667598 2311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:14.671099 2311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:34021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=37587
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.2311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:14.671952 2311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:14.672997 2311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:14.681293 2319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.682052 2317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:14.682780 2316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:14.683539 2311 server_base.cc:1034] running on GCE node
I20240503 01:59:14.683908 2311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:14.684527 2311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:14.685672 2311 hybrid_clock.cc:648] HybridClock initialized: now 1714701554685655 us; error 25 us; skew 500 ppm
I20240503 01:59:14.687534 2311 webserver.cc:460] Webserver started at http://127.0.99.136:37587/ using document root <none> and password file <none>
I20240503 01:59:14.688133 2311 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:14.688213 2311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:14.691654 2311 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:14.693526 2325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.694540 2311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:14.694651 2311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b6db6ee3749340658049b2301a938993"
format_stamp: "Formatted at 2024-05-03 01:59:07 on dist-test-slave-fvr5"
I20240503 01:59:14.695066 2311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:14.707481 2311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:14.708132 2311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:14.708276 2311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:14.708761 2311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:14.709584 2311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:14.709630 2311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.709666 2311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:14.709687 2311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:14.718472 2311 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34021
I20240503 01:59:14.718487 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34021 every 8 connection(s)
I20240503 01:59:14.719512 2311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:14.727563 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 2311
I20240503 01:59:14.728219 2439 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 01:59:14.728478 2439 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:14.729116 2439 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 01:59:14.730175 436 ts_manager.cc:194] Re-registered known tserver with Master: b6db6ee3749340658049b2301a938993 (127.0.99.136:34021)
I20240503 01:59:14.730940 436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:39721
I20240503 01:59:14.928027 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:14.931115 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.932698 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.933243 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:14.940987 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:14.941125 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.948185 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:14.967911 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.134368 433 ts_manager.cc:292] Set tserver state for 0a141082e3c24126a1542e2c9a3aad0e to MAINTENANCE_MODE
I20240503 01:59:15.207113 2046 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
W20240503 01:59:15.255945 2503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:15.256165 2503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:15.259223 2503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:15.347040 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.355319 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.356073 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.358799 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.360677 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.360893 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.372061 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.373164 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.380060 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.380419 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.385346 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.387513 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.393285 2177 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:15.398535 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.426786 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.429041 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.429644 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.537380 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.540618 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.549050 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.551939 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.556074 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.562639 2308 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:15.564153 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.570664 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.582341 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 01:59:15.666584 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:15.666886 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:15.670493 2596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:15.702567 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.703158 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.703918 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.714357 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.714885 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.727988 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.732836 2439 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 01:59:15.740453 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.744246 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.787218 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.788365 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.791249 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.793305 844 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } }
I20240503 01:59:15.794811 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:15.795181 1702 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:15.795698 582 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:15.796801 1702 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:15.799331 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.808313 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:15.815131 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.815955 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.816471 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.832625 1702 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } }
I20240503 01:59:15.835546 432 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:15.838266 976 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } }
I20240503 01:59:15.845142 582 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } }
I20240503 01:59:15.852707 844 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:15.854097 582 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:15.854501 976 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:15.854802 2708 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:15.854972 2704 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:15.856732 1702 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:15.857277 976 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:15.857365 582 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:15.859597 436 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 4 to 5, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:15.859597 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 83a04a16e66b463b98976c490155139c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 01:59:15.866226 779 consensus_peers.cc:485] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 -> Peer b6db6ee3749340658049b2301a938993 (127.0.99.136:34021): Couldn't send request to peer b6db6ee3749340658049b2301a938993. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83a04a16e66b463b98976c490155139c. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:15.994108 2718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:15.994302 2718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:15.997087 2718 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:16.073877 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.075590 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:16.080718 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.084491 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:16.087122 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.089262 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.096060 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.097729 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 01:59:16.256537 2760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:16.256770 2760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:16.259794 2760 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:16.268275 2769 ts_tablet_manager.cc:902] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:16.268997 2769 tablet_copy_client.cc:308] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:16.273222 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 83a04a16e66b463b98976c490155139c from peer b6db6ee3749340658049b2301a938993 ({username='slave'} at 127.0.99.136:50053)
I20240503 01:59:16.273380 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 83a04a16e66b463b98976c490155139c from peer b6db6ee3749340658049b2301a938993 at {username='slave'} at 127.0.99.136:50053: session id = b6db6ee3749340658049b2301a938993-83a04a16e66b463b98976c490155139c
I20240503 01:59:16.274541 865 tablet_copy_source_session.cc:205] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:16.277871 2769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:16.283674 2769 tablet_copy_client.cc:783] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:16.283895 2769 tablet_copy_client.cc:647] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:16.285243 2774 tablet_copy_client.cc:1002] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:16.339943 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.342921 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.349784 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:16.352738 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.360908 561 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:16.361142 956 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:16.361260 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:16.363929 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 01:59:16.414050 2501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 501
W20240503 01:59:16.425310 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:16.426021 780 consensus_peers.cc:485] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.566102 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.658794 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:16.659595 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.674350 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.740779 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.818284 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:16.819146 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:16.948186 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:17.356253 2831 raft_consensus.cc:482] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.356384 2831 raft_consensus.cc:504] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.356791 2831 leader_election.cc:288] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.357235 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
W20240503 01:59:17.357939 911 leader_election.cc:334] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.358008 911 leader_election.cc:302] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f; no voters: 0a141082e3c24126a1542e2c9a3aad0e, efd7ce56deab4305a002577febb80532
I20240503 01:59:17.358136 2831 raft_consensus.cc:2726] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:17.408208 1574 raft_consensus.cc:482] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.408357 1574 raft_consensus.cc:504] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:17.408811 1574 leader_election.cc:288] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.409257 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
W20240503 01:59:17.410022 649 leader_election.cc:334] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.410089 649 leader_election.cc:302] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efd7ce56deab4305a002577febb80532; no voters: 0a141082e3c24126a1542e2c9a3aad0e, c0801789471740eda4246b1ab4904e10
I20240503 01:59:17.410228 1574 raft_consensus.cc:2726] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:17.429967 2708 raft_consensus.cc:482] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.430099 2708 raft_consensus.cc:504] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.430473 2708 leader_election.cc:288] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.430910 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
W20240503 01:59:17.431555 780 leader_election.cc:334] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.431614 780 leader_election.cc:302] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f
I20240503 01:59:17.431782 2708 raft_consensus.cc:2726] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:17.513479 2708 raft_consensus.cc:482] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.513590 2708 raft_consensus.cc:504] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:17.513999 2708 leader_election.cc:288] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:17.514418 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
I20240503 01:59:17.514565 713 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 1.
I20240503 01:59:17.514856 778 leader_election.cc:302] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
W20240503 01:59:17.514997 780 leader_election.cc:334] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.515025 2708 raft_consensus.cc:2781] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:17.515072 2708 raft_consensus.cc:482] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.515122 2708 raft_consensus.cc:3037] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:17.515996 2708 raft_consensus.cc:504] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:17.516284 2708 leader_election.cc:288] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:17.516615 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f75e2ba8c494fc5bde411b10b687860" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:17.516749 713 raft_consensus.cc:3037] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.517185 780 leader_election.cc:334] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.517647 713 raft_consensus.cc:2454] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 2.
I20240503 01:59:17.517910 778 leader_election.cc:302] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.518091 2708 raft_consensus.cc:2781] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:17.518182 2708 raft_consensus.cc:686] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:17.518334 2708 consensus_queue.cc:237] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 01:59:17.519870 436 catalog_manager.cc:5484] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 1 to 2, leader changed from 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:17.560997 2846 raft_consensus.cc:482] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.561151 2846 raft_consensus.cc:504] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.561659 2846 leader_election.cc:288] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.562127 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
I20240503 01:59:17.562271 975 raft_consensus.cc:2454] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:17.562569 650 leader_election.cc:302] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:17.562731 2846 raft_consensus.cc:2781] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:17.562784 2846 raft_consensus.cc:482] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
W20240503 01:59:17.562825 649 leader_election.cc:334] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.562831 2846 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:17.563633 2846 raft_consensus.cc:504] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.563969 2846 leader_election.cc:288] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.564258 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481dd137b628482b9f1333aceb89ae39" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:17.564355 975 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.565107 649 leader_election.cc:334] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.565443 975 raft_consensus.cc:2454] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 2.
I20240503 01:59:17.565768 650 leader_election.cc:302] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, efd7ce56deab4305a002577febb80532; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.565965 2846 raft_consensus.cc:2781] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:17.566066 2846 raft_consensus.cc:686] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Becoming Leader. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:17.566200 2846 consensus_queue.cc:237] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.567715 436 catalog_manager.cc:5484] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 reported cstate change: term changed from 1 to 2, leader changed from 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) to efd7ce56deab4305a002577febb80532 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:17.706591 2708 raft_consensus.cc:482] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.706714 2708 raft_consensus.cc:504] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.707082 2708 leader_election.cc:288] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.707484 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f" is_pre_election: true
W20240503 01:59:17.708282 780 leader_election.cc:334] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.708362 780 leader_election.cc:302] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e, 648ac6ef81784cf3a916a2ed46655b3f
I20240503 01:59:17.708510 2708 raft_consensus.cc:2726] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:17.762948 2708 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.763072 2708 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:17.763432 2708 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.763847 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efd7ce56deab4305a002577febb80532" is_pre_election: true
W20240503 01:59:17.764503 780 leader_election.cc:334] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.764565 780 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e, efd7ce56deab4305a002577febb80532
I20240503 01:59:17.764675 2708 raft_consensus.cc:2726] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 01:59:17.777346 2832 raft_consensus.cc:482] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.777462 2832 raft_consensus.cc:504] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.777875 2832 leader_election.cc:288] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
W20240503 01:59:17.779330 911 leader_election.cc:334] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.781518 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:17.781659 844 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:17.781904 912 leader_election.cc:302] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.782059 2832 raft_consensus.cc:2781] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:17.782117 2832 raft_consensus.cc:482] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.782171 2832 raft_consensus.cc:3037] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:17.782971 2832 raft_consensus.cc:504] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.783272 2832 leader_election.cc:288] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.783550 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835fae0d5049f1a7e31fa51fd58f7a" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:17.783668 844 raft_consensus.cc:3037] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.784109 911 leader_election.cc:334] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.784569 844 raft_consensus.cc:2454] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 2.
I20240503 01:59:17.784814 912 leader_election.cc:302] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.784958 2832 raft_consensus.cc:2781] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:17.785041 2832 raft_consensus.cc:686] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:17.785184 2832 consensus_queue.cc:237] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.785673 2769 tablet_copy_client.cc:523] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:17.787024 432 catalog_manager.cc:5484] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 1 to 2, leader changed from 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:17.787726 2769 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Bootstrap starting.
I20240503 01:59:17.793133 2769 log.cc:824] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:17.812108 2769 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:17.812389 2769 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Bootstrap complete.
I20240503 01:59:17.812630 2769 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.007s
I20240503 01:59:17.814437 2769 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:17.814641 2769 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6db6ee3749340658049b2301a938993, State: Initialized, Role: LEARNER
I20240503 01:59:17.814903 2769 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:17.815675 2769 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:17.816438 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session b6db6ee3749340658049b2301a938993-83a04a16e66b463b98976c490155139c received from {username='slave'} at 127.0.99.136:50053
I20240503 01:59:17.816540 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session b6db6ee3749340658049b2301a938993-83a04a16e66b463b98976c490155139c on tablet 83a04a16e66b463b98976c490155139c with peer b6db6ee3749340658049b2301a938993
I20240503 01:59:17.822237 2832 raft_consensus.cc:482] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.822336 2832 raft_consensus.cc:504] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.822713 2832 leader_election.cc:288] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.823038 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:17.823184 844 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 1.
I20240503 01:59:17.823451 912 leader_election.cc:302] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:17.823658 2832 raft_consensus.cc:2781] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:17.823724 2832 raft_consensus.cc:482] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.823801 2832 raft_consensus.cc:3037] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.823828 911 leader_election.cc:334] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.824576 2832 raft_consensus.cc:504] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.824846 2832 leader_election.cc:288] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 01:59:17.825160 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae517688d34428e9a4983829376002b" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:17.825277 844 raft_consensus.cc:3037] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.825726 911 leader_election.cc:334] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.826202 844 raft_consensus.cc:2454] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 2.
I20240503 01:59:17.826444 912 leader_election.cc:302] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.826591 2832 raft_consensus.cc:2781] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:17.826675 2832 raft_consensus.cc:686] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:17.826804 2832 consensus_queue.cc:237] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 01:59:17.828213 432 catalog_manager.cc:5484] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 1 to 2, leader changed from 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:17.850749 2846 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.850872 2846 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:17.851265 2846 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:17.851629 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10" is_pre_election: true
I20240503 01:59:17.851807 844 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 1.
I20240503 01:59:17.852106 650 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:17.852257 2846 raft_consensus.cc:2781] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 01:59:17.852304 2846 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a141082e3c24126a1542e2c9a3aad0e)
I20240503 01:59:17.852336 2846 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:17.853091 2846 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:17.853338 2846 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: Requested vote from peers 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079), c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
W20240503 01:59:17.853442 649 leader_election.cc:334] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.853652 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "efd7ce56deab4305a002577febb80532" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:17.853767 844 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
W20240503 01:59:17.854403 649 leader_election.cc:334] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:17.854614 844 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7ce56deab4305a002577febb80532 in term 2.
I20240503 01:59:17.854843 650 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters: 0a141082e3c24126a1542e2c9a3aad0e
I20240503 01:59:17.855000 2846 raft_consensus.cc:2781] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:17.855088 2846 raft_consensus.cc:686] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Becoming Leader. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:17.855222 2846 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:17.856762 432 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 reported cstate change: term changed from 1 to 2, leader changed from 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) to efd7ce56deab4305a002577febb80532 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:18.050921 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.053488 713 raft_consensus.cc:1261] T 4f75e2ba8c494fc5bde411b10b687860 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:18.053896 2708 consensus_queue.cc:1035] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:18.141448 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.166733 975 raft_consensus.cc:1261] T 481dd137b628482b9f1333aceb89ae39 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:18.167191 2846 consensus_queue.cc:1035] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:18.231187 844 raft_consensus.cc:1261] T e6835fae0d5049f1a7e31fa51fd58f7a P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:18.231664 2832 consensus_queue.cc:1035] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:18.234830 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:18.236635 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:18.260006 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.273788 2393 raft_consensus.cc:1203] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 01:59:18.324872 844 raft_consensus.cc:1261] T bae517688d34428e9a4983829376002b P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:18.325313 2832 consensus_queue.cc:1035] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:18.415305 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 01:59:18.415793 2846 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:18.719789 2901 raft_consensus.cc:1050] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER b6db6ee3749340658049b2301a938993 to VOTER
I20240503 01:59:18.720304 2901 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:18.721674 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:18.722002 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:18.722227 2903 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:18.722455 2901 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:18.722743 780 consensus_peers.cc:485] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:18.725382 2901 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:18.725903 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:18.727106 975 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:18.727857 432 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 5 to 6, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:18.730912 844 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:18.732066 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:18.732370 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:18.732420 2901 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:18.732730 2901 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:18.733820 2708 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:18.734259 975 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:18.734997 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83a04a16e66b463b98976c490155139c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:18.735770 437 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:18.736205 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
W20240503 01:59:18.738564 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:18.739408 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83a04a16e66b463b98976c490155139c: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:19.019805 713 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:19.021075 844 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:19.021471 2923 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:19.021699 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:19.023128 2894 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:19.023492 844 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:19.025905 436 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:19.029034 713 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:19.030473 844 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:19.031245 2894 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:19.031548 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:19.032886 2925 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:19.033212 844 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:19.034363 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:19.035130 432 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:19.036001 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505): Couldn't send request to peer 091deff9eeda46b4ab93b05a3f466847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42cb1c96b35a4b98a1edfa4ccd99602d. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:19.072419 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:19.102571 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:19.172076 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:19.222389 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:19.304891 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:19.529408 2943 ts_tablet_manager.cc:902] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:19.530174 2943 tablet_copy_client.cc:308] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:19.533639 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer 091deff9eeda46b4ab93b05a3f466847 ({username='slave'} at 127.0.99.133:40197)
I20240503 01:59:19.533768 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer 091deff9eeda46b4ab93b05a3f466847 at {username='slave'} at 127.0.99.133:40197: session id = 091deff9eeda46b4ab93b05a3f466847-42cb1c96b35a4b98a1edfa4ccd99602d
I20240503 01:59:19.534858 734 tablet_copy_source_session.cc:205] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:19.538151 2943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:19.543247 2943 tablet_copy_client.cc:783] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:19.543480 2943 tablet_copy_client.cc:647] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:19.544979 2947 tablet_copy_client.cc:1002] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:19.949141 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83a04a16e66b463b98976c490155139c: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:20.280746 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:20.301959 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:20.528882 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:20.654040 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:20.806214 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:21.017033 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83a04a16e66b463b98976c490155139c: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:21.045408 2943 tablet_copy_client.cc:523] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:21.047396 2943 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Bootstrap starting.
I20240503 01:59:21.052387 2943 log.cc:824] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:21.072122 2943 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:21.072412 2943 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Bootstrap complete.
I20240503 01:59:21.072659 2943 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.004s
I20240503 01:59:21.074586 2943 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:21.074790 2943 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Initialized, Role: LEARNER
I20240503 01:59:21.075058 2943 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:21.075826 2943 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:21.076543 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session 091deff9eeda46b4ab93b05a3f466847-42cb1c96b35a4b98a1edfa4ccd99602d received from {username='slave'} at 127.0.99.133:40197
I20240503 01:59:21.076646 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session 091deff9eeda46b4ab93b05a3f466847-42cb1c96b35a4b98a1edfa4ccd99602d on tablet 42cb1c96b35a4b98a1edfa4ccd99602d with peer 091deff9eeda46b4ab93b05a3f466847
W20240503 01:59:21.395890 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:21.617237 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 01:59:21.617874 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 01:59:21.626289 2000 raft_consensus.cc:1203] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 01:59:21.629410 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 01:59:21.759965 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 01:59:21.760581 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:21.800662 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:21.943087 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:21.989342 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 01:59:22.211126 3026 raft_consensus.cc:1050] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER 091deff9eeda46b4ab93b05a3f466847 to VOTER
I20240503 01:59:22.211591 3026 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:22.213066 844 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:22.213333 2000 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:22.213517 3026 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:22.213764 3026 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:22.214485 649 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:22.216524 3029 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.217016 2000 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.218297 844 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.218823 436 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.222049 713 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:22.223305 844 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:22.223361 2000 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:22.223639 3029 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:22.223863 2979 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:22.225167 3029 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.225610 844 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.226382 2000 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:22.226776 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:22.227680 432 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 01:59:22.231017 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 42cb1c96b35a4b98a1edfa4ccd99602d: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:22.456478 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 01:59:22.544857 713 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:22.546358 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:22.546936 2979 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:22.548074 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:22.548892 2979 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:22.551235 844 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:22.551811 436 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:22.555928 713 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:22.557473 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:22.557898 3028 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:22.558732 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:22.559484 3029 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:22.559896 844 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:22.561993 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba2b3e448315456789bfa5953e80c7ce with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:22.562613 432 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 5 to 6, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
W20240503 01:59:22.563354 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873): Couldn't send request to peer e139f2784fa5493983f786b885d7c33d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba2b3e448315456789bfa5953e80c7ce. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:22.740267 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:23.032346 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 01:59:23.035307 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 01:59:23.174392 3069 ts_tablet_manager.cc:902] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:23.175136 3069 tablet_copy_client.cc:308] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:23.179158 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet ba2b3e448315456789bfa5953e80c7ce from peer e139f2784fa5493983f786b885d7c33d ({username='slave'} at 127.0.99.134:59511)
I20240503 01:59:23.179267 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet ba2b3e448315456789bfa5953e80c7ce from peer e139f2784fa5493983f786b885d7c33d at {username='slave'} at 127.0.99.134:59511: session id = e139f2784fa5493983f786b885d7c33d-ba2b3e448315456789bfa5953e80c7ce
I20240503 01:59:23.192485 734 tablet_copy_source_session.cc:205] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:23.196123 3069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b3e448315456789bfa5953e80c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.202598 3069 tablet_copy_client.cc:783] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:23.202811 3069 tablet_copy_client.cc:647] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:23.204072 3072 tablet_copy_client.cc:1002] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:23.371109 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 42cb1c96b35a4b98a1edfa4ccd99602d: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:23.734262 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:23.949127 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:24.384980 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:24.427068 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:24.436100 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 01:59:24.436733 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 42cb1c96b35a4b98a1edfa4ccd99602d: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:24.548252 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:24.704490 3069 tablet_copy_client.cc:523] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:24.706394 3069 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Bootstrap starting.
I20240503 01:59:24.712111 3069 log.cc:824] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:24.731945 3069 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:24.732234 3069 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Bootstrap complete.
I20240503 01:59:24.732481 3069 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.005s
I20240503 01:59:24.734361 3069 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:24.734560 3069 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Initialized, Role: LEARNER
I20240503 01:59:24.734831 3069 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:24.735500 3069 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:24.736158 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session e139f2784fa5493983f786b885d7c33d-ba2b3e448315456789bfa5953e80c7ce received from {username='slave'} at 127.0.99.134:59511
I20240503 01:59:24.736236 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session e139f2784fa5493983f786b885d7c33d-ba2b3e448315456789bfa5953e80c7ce on tablet ba2b3e448315456789bfa5953e80c7ce with peer e139f2784fa5493983f786b885d7c33d
W20240503 01:59:24.820022 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:25.128005 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 01:59:25.150920 2131 raft_consensus.cc:1203] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 01:59:25.290494 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:25.332979 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:25.733873 3150 raft_consensus.cc:1050] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER e139f2784fa5493983f786b885d7c33d to VOTER
I20240503 01:59:25.734546 3150 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:25.736143 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:25.736517 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:25.736529 3150 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:25.737237 3150 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 01:59:25.737759 649 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:25.739816 3152 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:25.740403 2131 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:25.742005 844 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:25.742523 432 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 6 to 7, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:25.750649 714 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:25.752028 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:25.752414 3150 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:25.752542 844 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:25.752943 3150 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:25.754364 3094 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:25.754968 2131 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:25.755744 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba2b3e448315456789bfa5953e80c7ce with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:25.756644 437 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 7 to 8, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:25.757736 844 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
W20240503 01:59:25.760535 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 01:59:25.762298 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba2b3e448315456789bfa5953e80c7ce: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:26.069552 844 consensus_queue.cc:237] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 01:59:26.070768 714 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:26.071146 2708 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:26.071727 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:26.073169 3174 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:26.073611 714 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } }
I20240503 01:59:26.075804 432 catalog_manager.cc:5484] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:26.078784 844 consensus_queue.cc:237] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:26.080507 714 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:26.080893 3174 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:26.082429 3176 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:26.082834 714 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
W20240503 01:59:26.083062 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:26.084913 436 catalog_manager.cc:5484] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 4 to 5, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
W20240503 01:59:26.085837 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505): Couldn't send request to peer 091deff9eeda46b4ab93b05a3f466847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8018cb49863d4736ae5f79f472f63bb8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:26.104866 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8018cb49863d4736ae5f79f472f63bb8 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 01:59:26.172096 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 01:59:26.535399 3197 ts_tablet_manager.cc:902] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:26.535892 3197 tablet_copy_client.cc:308] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:26.539041 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 8018cb49863d4736ae5f79f472f63bb8 from peer 091deff9eeda46b4ab93b05a3f466847 ({username='slave'} at 127.0.99.133:45561)
I20240503 01:59:26.539136 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 8018cb49863d4736ae5f79f472f63bb8 from peer 091deff9eeda46b4ab93b05a3f466847 at {username='slave'} at 127.0.99.133:45561: session id = 091deff9eeda46b4ab93b05a3f466847-8018cb49863d4736ae5f79f472f63bb8
I20240503 01:59:26.539932 865 tablet_copy_source_session.cc:205] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:26.540845 3197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8018cb49863d4736ae5f79f472f63bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.542534 3197 tablet_copy_client.cc:783] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:26.542688 3197 tablet_copy_client.cc:647] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:26.543655 3201 tablet_copy_client.cc:1002] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:26.618420 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 01:59:26.815716 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 01:59:26.816380 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:26.863653 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:26.913357 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba2b3e448315456789bfa5953e80c7ce: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:27.081812 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 01:59:27.308892 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:27.371230 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:27.657831 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:27.968631 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba2b3e448315456789bfa5953e80c7ce: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:27.991163 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 01:59:28.043991 3197 tablet_copy_client.cc:523] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:28.046466 3197 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap starting.
I20240503 01:59:28.064687 3197 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:28.065105 3197 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap complete.
I20240503 01:59:28.065282 3197 ts_tablet_manager.cc:1367] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240503 01:59:28.065763 3197 raft_consensus.cc:348] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:28.065869 3197 raft_consensus.cc:729] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Initialized, Role: LEARNER
I20240503 01:59:28.065964 3197 consensus_queue.cc:260] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:28.066264 3197 ts_tablet_manager.cc:1398] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:28.066743 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session 091deff9eeda46b4ab93b05a3f466847-8018cb49863d4736ae5f79f472f63bb8 received from {username='slave'} at 127.0.99.133:45561
I20240503 01:59:28.066814 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session 091deff9eeda46b4ab93b05a3f466847-8018cb49863d4736ae5f79f472f63bb8 on tablet 8018cb49863d4736ae5f79f472f63bb8 with peer 091deff9eeda46b4ab93b05a3f466847
W20240503 01:59:28.276633 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 01:59:28.395687 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:28.501313 1999 raft_consensus.cc:1203] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 01:59:28.753765 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 01:59:29.104337 2708 raft_consensus.cc:1050] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER 091deff9eeda46b4ab93b05a3f466847 to VOTER
I20240503 01:59:29.104789 2708 consensus_queue.cc:237] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:29.106227 1999 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:29.106704 714 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:29.106926 3277 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:29.107118 780 consensus_peers.cc:485] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:29.107159 3278 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:29.108971 3277 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.109457 714 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.111275 1999 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.111397 436 catalog_manager.cc:5484] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 5 to 6, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.114873 844 consensus_queue.cc:237] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:29.115892 1999 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:29.115988 714 raft_consensus.cc:1261] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:29.116209 3277 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:29.116362 3275 consensus_queue.cc:1035] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:29.117623 3277 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.118088 1999 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.118186 714 raft_consensus.cc:2932] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:29.119174 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8018cb49863d4736ae5f79f472f63bb8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:29.120352 432 catalog_manager.cc:5484] T 8018cb49863d4736ae5f79f472f63bb8 P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
W20240503 01:59:29.123366 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:29.252332 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 01:59:29.410686 975 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 01:59:29.411944 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:29.412279 3301 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:29.412444 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:29.413798 2985 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } }
I20240503 01:59:29.414223 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } }
I20240503 01:59:29.416667 437 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } }
I20240503 01:59:29.419382 975 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:29.420650 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:29.421027 3301 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 01:59:29.421871 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:29.422674 3308 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:29.423004 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:29.424243 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b0c46bbf4f1b435993ce1057b2b4de9e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 01:59:29.424763 436 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 4 to 5, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
W20240503 01:59:29.426360 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873): Couldn't send request to peer e139f2784fa5493983f786b885d7c33d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0c46bbf4f1b435993ce1057b2b4de9e. This is attempt 1: this message will repeat every 5th retry.
W20240503 01:59:29.451701 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240503 01:59:29.452297 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:29.512547 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 01:59:29.875941 3323 ts_tablet_manager.cc:902] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Initiating tablet copy from peer 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:29.876442 3323 tablet_copy_client.cc:308] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34053
I20240503 01:59:29.880188 996 tablet_copy_service.cc:140] P 648ac6ef81784cf3a916a2ed46655b3f: Received BeginTabletCopySession request for tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer e139f2784fa5493983f786b885d7c33d ({username='slave'} at 127.0.99.134:48149)
I20240503 01:59:29.880326 996 tablet_copy_service.cc:161] P 648ac6ef81784cf3a916a2ed46655b3f: Beginning new tablet copy session on tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer e139f2784fa5493983f786b885d7c33d at {username='slave'} at 127.0.99.134:48149: session id = e139f2784fa5493983f786b885d7c33d-b0c46bbf4f1b435993ce1057b2b4de9e
I20240503 01:59:29.881542 996 tablet_copy_source_session.cc:205] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:29.882614 3323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.884431 3323 tablet_copy_client.cc:783] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:29.884622 3323 tablet_copy_client.cc:647] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:29.885790 3327 tablet_copy_client.cc:1002] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:29.987406 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:30.253604 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:30.622385 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:30.984792 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 01:59:31.240520 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:31.296360 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:31.386127 3323 tablet_copy_client.cc:523] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:31.387825 3323 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Bootstrap starting.
I20240503 01:59:31.406363 3323 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:31.406631 3323 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Bootstrap complete.
I20240503 01:59:31.406847 3323 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20240503 01:59:31.407363 3323 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:31.407480 3323 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Initialized, Role: LEARNER
I20240503 01:59:31.407575 3323 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:31.407982 3323 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Time spent starting tablet: real 0.001s user 0.001s sys 0.003s
I20240503 01:59:31.408566 996 tablet_copy_service.cc:342] P 648ac6ef81784cf3a916a2ed46655b3f: Request end of tablet copy session e139f2784fa5493983f786b885d7c33d-b0c46bbf4f1b435993ce1057b2b4de9e received from {username='slave'} at 127.0.99.134:48149
I20240503 01:59:31.408658 996 tablet_copy_service.cc:433] P 648ac6ef81784cf3a916a2ed46655b3f: ending tablet copy session e139f2784fa5493983f786b885d7c33d-b0c46bbf4f1b435993ce1057b2b4de9e on tablet b0c46bbf4f1b435993ce1057b2b4de9e with peer e139f2784fa5493983f786b885d7c33d
I20240503 01:59:31.437160 2131 raft_consensus.cc:1203] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 01:59:31.829690 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 01:59:31.830345 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:31.872367 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 01:59:31.936146 3389 raft_consensus.cc:1050] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: attempting to promote NON_VOTER e139f2784fa5493983f786b885d7c33d to VOTER
I20240503 01:59:31.936551 3389 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:31.937793 2131 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:31.937865 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:31.938292 3388 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:31.938421 3391 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 01:59:31.938784 911 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:31.940130 3388 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:31.940618 2131 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:31.942945 436 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 5 to 6, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:31.944391 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
W20240503 01:59:31.944826 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 26 similar messages]
I20240503 01:59:31.949348 975 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:31.950448 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:31.950700 2131 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:31.950779 3391 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:31.951064 3391 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:31.952181 3340 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:31.952582 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:31.953758 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0c46bbf4f1b435993ce1057b2b4de9e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:31.954416 2131 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:31.954514 437 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 6 to 7, VOTER 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:32.074970 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:32.156427 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:32.309062 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0c46bbf4f1b435993ce1057b2b4de9e: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:32.333415 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 01:59:32.360342 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.360760 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:32.362659 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.364356 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.365161 824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:32.390635 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.401715 694 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 01:59:32.429057 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 01:59:32.483446 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.484489 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.487597 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.488790 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:32.489742 824 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:32.506280 694 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:32.507227 2111 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:32.555895 845 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:32.558298 975 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:32.558395 714 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:32.558889 3490 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:32.560014 3275 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:32.567413 3275 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:32.570024 975 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
W20240503 01:59:32.570746 781 consensus_peers.cc:485] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 -> Peer 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161): Couldn't send request to peer 07d11a6a41ef47b888dae403ffbf6456. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4266aabe93664eb889ebf7cca050e3c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.570618 433 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.571069 714 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:32.579710 975 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:32.581352 2131 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:32.581599 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:32.581887 3502 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:32.582139 3501 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:32.583662 3391 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:32.585523 2131 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
W20240503 01:59:32.585767 910 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f -> Peer b6db6ee3749340658049b2301a938993 (127.0.99.136:34021): Couldn't send request to peer b6db6ee3749340658049b2301a938993. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0c46bbf4f1b435993ce1057b2b4de9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:32.586099 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:32.587344 440 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 7 to 8, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:32.974292 3526 ts_tablet_manager.cc:902] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:32.975003 3526 tablet_copy_client.cc:308] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:32.978544 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 4266aabe93664eb889ebf7cca050e3c2 from peer 07d11a6a41ef47b888dae403ffbf6456 ({username='slave'} at 127.0.99.135:35807)
I20240503 01:59:32.978641 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 4266aabe93664eb889ebf7cca050e3c2 from peer 07d11a6a41ef47b888dae403ffbf6456 at {username='slave'} at 127.0.99.135:35807: session id = 07d11a6a41ef47b888dae403ffbf6456-4266aabe93664eb889ebf7cca050e3c2
I20240503 01:59:32.979394 865 tablet_copy_source_session.cc:205] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:32.982352 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.987697 3526 tablet_copy_client.cc:783] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:32.987995 3526 tablet_copy_client.cc:647] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:32.989272 3530 tablet_copy_client.cc:1002] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:33.117991 3542 ts_tablet_manager.cc:902] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Initiating tablet copy from peer 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:33.118464 3542 tablet_copy_client.cc:308] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34053
I20240503 01:59:33.121768 996 tablet_copy_service.cc:140] P 648ac6ef81784cf3a916a2ed46655b3f: Received BeginTabletCopySession request for tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer b6db6ee3749340658049b2301a938993 ({username='slave'} at 127.0.99.136:52873)
I20240503 01:59:33.121879 996 tablet_copy_service.cc:161] P 648ac6ef81784cf3a916a2ed46655b3f: Beginning new tablet copy session on tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer b6db6ee3749340658049b2301a938993 at {username='slave'} at 127.0.99.136:52873: session id = b6db6ee3749340658049b2301a938993-b0c46bbf4f1b435993ce1057b2b4de9e
I20240503 01:59:33.122906 996 tablet_copy_source_session.cc:205] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:33.123745 3542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.125528 3542 tablet_copy_client.cc:783] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:33.125672 3542 tablet_copy_client.cc:647] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:33.127007 3546 tablet_copy_client.cc:1002] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:33.175180 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:33.323415 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 01:59:33.352242 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:33.468678 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:34.082283 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:34.327008 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 01:59:34.440948 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 01:59:34.489737 3526 tablet_copy_client.cc:523] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:34.491835 3526 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap starting.
I20240503 01:59:34.497076 3526 log.cc:824] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:34.523455 3526 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:34.523882 3526 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap complete.
I20240503 01:59:34.524204 3526 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.008s
I20240503 01:59:34.526227 3526 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:34.526466 3526 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07d11a6a41ef47b888dae403ffbf6456, State: Initialized, Role: LEARNER
I20240503 01:59:34.526904 3526 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:34.527977 3526 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 01:59:34.528637 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session 07d11a6a41ef47b888dae403ffbf6456-4266aabe93664eb889ebf7cca050e3c2 received from {username='slave'} at 127.0.99.135:35807
I20240503 01:59:34.528749 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session 07d11a6a41ef47b888dae403ffbf6456-4266aabe93664eb889ebf7cca050e3c2 on tablet 4266aabe93664eb889ebf7cca050e3c2 with peer 07d11a6a41ef47b888dae403ffbf6456
I20240503 01:59:34.602931 2262 raft_consensus.cc:1203] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:34.627326 3542 tablet_copy_client.cc:523] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:34.628775 3542 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Bootstrap starting.
I20240503 01:59:34.647069 3542 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:34.647341 3542 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Bootstrap complete.
I20240503 01:59:34.647497 3542 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 01:59:34.648017 3542 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:34.648152 3542 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6db6ee3749340658049b2301a938993, State: Initialized, Role: LEARNER
I20240503 01:59:34.648260 3542 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:34.648619 3542 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 01:59:34.649158 996 tablet_copy_service.cc:342] P 648ac6ef81784cf3a916a2ed46655b3f: Request end of tablet copy session b6db6ee3749340658049b2301a938993-b0c46bbf4f1b435993ce1057b2b4de9e received from {username='slave'} at 127.0.99.136:52873
I20240503 01:59:34.649250 996 tablet_copy_service.cc:433] P 648ac6ef81784cf3a916a2ed46655b3f: ending tablet copy session b6db6ee3749340658049b2301a938993-b0c46bbf4f1b435993ce1057b2b4de9e on tablet b0c46bbf4f1b435993ce1057b2b4de9e with peer b6db6ee3749340658049b2301a938993
I20240503 01:59:34.808035 2393 raft_consensus.cc:1203] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 01:59:34.908828 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 01:59:35.163583 3608 raft_consensus.cc:1050] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 to VOTER
I20240503 01:59:35.164093 3608 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:35.165534 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:35.165529 975 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:35.165694 714 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:35.166083 3674 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:35.166332 3673 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:35.166550 3671 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:35.169107 3671 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:35.169561 714 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:35.169855 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:35.171401 975 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:35.172351 432 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:35.312716 3683 raft_consensus.cc:1050] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: attempting to promote NON_VOTER b6db6ee3749340658049b2301a938993 to VOTER
I20240503 01:59:35.313095 3683 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:35.314589 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:35.314633 714 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:35.314793 2131 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:35.314952 3683 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:35.315171 3622 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:35.315377 3684 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:35.317181 3683 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:35.317864 2393 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:35.319104 714 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:35.319842 440 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 8 to 9, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:35.320657 2131 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:35.361274 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e"
dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53482
I20240503 01:59:35.361446 975 raft_consensus.cc:595] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:35.371050 845 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35210
I20240503 01:59:35.371205 845 raft_consensus.cc:595] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Received request to transfer leadership
W20240503 01:59:35.444476 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 01:59:35.445259 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83a04a16e66b463b98976c490155139c: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:35.574074 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 01:59:35.582556 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e"
dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53504
I20240503 01:59:35.582695 975 raft_consensus.cc:595] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:35.593273 845 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35228
I20240503 01:59:35.593405 845 raft_consensus.cc:595] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:35.647922 3608 raft_consensus.cc:979] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: : Instructing follower 648ac6ef81784cf3a916a2ed46655b3f to start an election
I20240503 01:59:35.648056 3608 raft_consensus.cc:1067] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Signalling peer 648ac6ef81784cf3a916a2ed46655b3f to start an election
I20240503 01:59:35.648566 975 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
from {username='slave'} at 127.0.99.131:54083
I20240503 01:59:35.648715 975 raft_consensus.cc:482] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:35.648784 975 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.649674 975 raft_consensus.cc:504] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:35.650235 975 leader_election.cc:288] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:35.650511 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:35.650576 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:35.650627 845 raft_consensus.cc:3032] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:35.650665 845 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:35.650713 713 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.650832 845 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:35.651051 845 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.651762 713 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 2.
I20240503 01:59:35.652076 845 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 2.
I20240503 01:59:35.652423 912 leader_election.cc:302] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:35.652614 3684 raft_consensus.cc:2781] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:35.652714 3684 raft_consensus.cc:686] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Becoming Leader. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:35.652849 3684 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:35.654737 440 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: term changed from 1 to 2, leader changed from c0801789471740eda4246b1ab4904e10 (127.0.99.131) to 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132). New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.655647 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "648ac6ef81784cf3a916a2ed46655b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 01:59:35.655848 2262 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.656667 2262 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648ac6ef81784cf3a916a2ed46655b3f in term 2.
I20240503 01:59:35.717878 3684 raft_consensus.cc:979] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: : Instructing follower e139f2784fa5493983f786b885d7c33d to start an election
I20240503 01:59:35.718008 3684 raft_consensus.cc:1067] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Signalling peer e139f2784fa5493983f786b885d7c33d to start an election
I20240503 01:59:35.718573 2131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e"
dest_uuid: "e139f2784fa5493983f786b885d7c33d"
from {username='slave'} at 127.0.99.132:47105
I20240503 01:59:35.718752 2131 raft_consensus.cc:482] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:35.718833 2131 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.719619 2131 raft_consensus.cc:504] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:35.720650 2131 leader_election.cc:288] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [CANDIDATE]: Term 2 election: Requested vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), b6db6ee3749340658049b2301a938993 (127.0.99.136:34021)
I20240503 01:59:35.720949 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:35.721086 713 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.721051 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 01:59:35.721158 975 raft_consensus.cc:3032] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:35.721196 975 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 01:59:35.721369 975 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:35.721556 975 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.722139 713 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 2.
I20240503 01:59:35.722584 975 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 2.
I20240503 01:59:35.722901 2067 leader_election.cc:302] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 648ac6ef81784cf3a916a2ed46655b3f, e139f2784fa5493983f786b885d7c33d, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:35.723134 3691 raft_consensus.cc:2781] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:35.723292 3691 raft_consensus.cc:686] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 LEADER]: Becoming Leader. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Running, Role: LEADER
I20240503 01:59:35.723568 3691 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:35.725905 440 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d reported cstate change: term changed from 1 to 2, leader changed from 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) to e139f2784fa5493983f786b885d7c33d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:35.726135 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0c46bbf4f1b435993ce1057b2b4de9e" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b6db6ee3749340658049b2301a938993"
I20240503 01:59:35.726298 2393 raft_consensus.cc:3037] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:35.727286 2393 raft_consensus.cc:2454] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 2.
W20240503 01:59:35.904548 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 01:59:35.937417 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 01:59:36.039973 713 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:36.040422 3684 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:36.125869 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:36.126334 3691 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:36.131731 845 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:36.132301 3684 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:36.134825 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:36.135736 3684 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:36.135931 975 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:36.136464 3691 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:36.139557 713 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:36.141026 3739 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:36.583786 975 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:36.585059 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:36.585059 713 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 01:59:36.585510 3684 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:36.585729 3763 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:36.587463 3622 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:36.587888 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:36.588973 713 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:36.589525 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4266aabe93664eb889ebf7cca050e3c2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:36.589860 437 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 5 to 7, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:36.592689 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 with delete_type TABLET_DATA_TOMBSTONED (TS c0801789471740eda4246b1ab4904e10 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57380
I20240503 01:59:36.593199 3797 tablet_replica.cc:330] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: stopping tablet replica
I20240503 01:59:36.593504 3797 raft_consensus.cc:2229] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.593670 3797 raft_consensus.cc:2258] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.594723 3797 ts_tablet_manager.cc:1875] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:36.596551 3797 ts_tablet_manager.cc:1888] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 01:59:36.596632 3797 log.cc:1196] T 4266aabe93664eb889ebf7cca050e3c2 P c0801789471740eda4246b1ab4904e10: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4266aabe93664eb889ebf7cca050e3c2
I20240503 01:59:36.597153 428 catalog_manager.cc:4830] TS c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797): tablet 4266aabe93664eb889ebf7cca050e3c2 (table rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]) successfully deleted
I20240503 01:59:36.658954 845 consensus_queue.cc:237] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:36.660423 975 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:36.660550 781 consensus_peers.cc:485] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 -> Peer 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161): Couldn't send request to peer 07d11a6a41ef47b888dae403ffbf6456. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd5de36fcad46a0bf6aca04394e46c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.660579 713 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.660789 3816 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.661015 3815 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.662673 3816 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:36.663072 975 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:36.663144 713 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:36.665371 433 catalog_manager.cc:5484] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index -1 to 4, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:36.674206 713 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
W20240503 01:59:36.675740 649 consensus_peers.cc:485] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 -> Peer e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873): Couldn't send request to peer e139f2784fa5493983f786b885d7c33d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a63bd458e2e94636a3a64ddc59ad8196. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.675971 845 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.676028 975 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.676333 3150 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.676537 3826 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.678159 3150 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:36.678539 975 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:36.680853 437 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index -1 to 4, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
W20240503 01:59:36.681336 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 01:59:36.682158 845 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:36.738327 2131 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:36.739463 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:36.739465 713 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:36.739845 3845 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:36.740047 3846 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:36.741559 3845 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:36.741976 2393 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:36.742488 713 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:36.744436 433 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 9 to 11, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:36.745196 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0c46bbf4f1b435993ce1057b2b4de9e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:36.747205 956 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c46bbf4f1b435993ce1057b2b4de9e with delete_type TABLET_DATA_TOMBSTONED (TS 648ac6ef81784cf3a916a2ed46655b3f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43320
I20240503 01:59:36.747741 3848 tablet_replica.cc:330] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: stopping tablet replica
I20240503 01:59:36.748051 3848 raft_consensus.cc:2229] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.748208 3848 raft_consensus.cc:2258] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.749141 3848 ts_tablet_manager.cc:1875] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:36.750839 3848 ts_tablet_manager.cc:1888] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 01:59:36.750911 3848 log.cc:1196] T b0c46bbf4f1b435993ce1057b2b4de9e P 648ac6ef81784cf3a916a2ed46655b3f: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0c46bbf4f1b435993ce1057b2b4de9e
I20240503 01:59:36.751421 428 catalog_manager.cc:4830] TS 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053): tablet b0c46bbf4f1b435993ce1057b2b4de9e (table rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]) successfully deleted
W20240503 01:59:36.871742 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 01:59:36.896733 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 01:59:36.926173 845 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:36.927563 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:36.927692 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:36.928030 3815 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 01:59:36.928192 780 consensus_peers.cc:485] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 -> Peer 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505): Couldn't send request to peer 091deff9eeda46b4ab93b05a3f466847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83a04a16e66b463b98976c490155139c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.928364 3815 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:36.930146 3608 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:36.932018 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:36.932544 440 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:36.938138 975 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
W20240503 01:59:36.966181 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 01:59:37.069059 3889 ts_tablet_manager.cc:902] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:37.069530 3889 tablet_copy_client.cc:308] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:37.070104 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet a63bd458e2e94636a3a64ddc59ad8196 from peer e139f2784fa5493983f786b885d7c33d ({username='slave'} at 127.0.99.134:59511)
I20240503 01:59:37.070212 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet a63bd458e2e94636a3a64ddc59ad8196 from peer e139f2784fa5493983f786b885d7c33d at {username='slave'} at 127.0.99.134:59511: session id = e139f2784fa5493983f786b885d7c33d-a63bd458e2e94636a3a64ddc59ad8196
I20240503 01:59:37.071166 734 tablet_copy_source_session.cc:205] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.072089 3889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63bd458e2e94636a3a64ddc59ad8196. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 01:59:37.073859 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 01:59:37.074210 3889 tablet_copy_client.cc:783] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.074355 3889 tablet_copy_client.cc:647] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.075264 3893 tablet_copy_client.cc:1002] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:37.087618 3894 ts_tablet_manager.cc:902] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:37.088361 3894 tablet_copy_client.cc:308] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:37.088837 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 3fd5de36fcad46a0bf6aca04394e46c7 from peer 07d11a6a41ef47b888dae403ffbf6456 ({username='slave'} at 127.0.99.135:35807)
I20240503 01:59:37.088953 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 3fd5de36fcad46a0bf6aca04394e46c7 from peer 07d11a6a41ef47b888dae403ffbf6456 at {username='slave'} at 127.0.99.135:35807: session id = 07d11a6a41ef47b888dae403ffbf6456-3fd5de36fcad46a0bf6aca04394e46c7
I20240503 01:59:37.089684 865 tablet_copy_source_session.cc:205] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.090484 3894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd5de36fcad46a0bf6aca04394e46c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.092206 3894 tablet_copy_client.cc:783] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.092342 3894 tablet_copy_client.cc:647] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.093291 3898 tablet_copy_client.cc:1002] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:37.152246 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 01:59:37.314025 3914 ts_tablet_manager.cc:902] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:37.314460 3914 tablet_copy_client.cc:308] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:37.314914 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 83a04a16e66b463b98976c490155139c from peer 091deff9eeda46b4ab93b05a3f466847 ({username='slave'} at 127.0.99.133:45561)
I20240503 01:59:37.315021 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 83a04a16e66b463b98976c490155139c from peer 091deff9eeda46b4ab93b05a3f466847 at {username='slave'} at 127.0.99.133:45561: session id = 091deff9eeda46b4ab93b05a3f466847-83a04a16e66b463b98976c490155139c
I20240503 01:59:37.315845 865 tablet_copy_source_session.cc:205] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.316640 3914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.318392 3914 tablet_copy_client.cc:783] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.318569 3914 tablet_copy_client.cc:647] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.319545 3918 tablet_copy_client.cc:1002] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:37.487486 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:37.564599 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 01:59:37.844516 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 01:59:38.548398 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 01:59:38.575601 3889 tablet_copy_client.cc:523] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.576994 3889 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Bootstrap starting.
I20240503 01:59:38.593614 3894 tablet_copy_client.cc:523] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.595067 3894 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap starting.
I20240503 01:59:38.597090 3889 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.597388 3889 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Bootstrap complete.
I20240503 01:59:38.597577 3889 ts_tablet_manager.cc:1367] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20240503 01:59:38.598037 3889 raft_consensus.cc:348] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:38.598153 3889 raft_consensus.cc:729] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Initialized, Role: LEARNER
I20240503 01:59:38.598268 3889 consensus_queue.cc:260] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:38.598711 3889 ts_tablet_manager.cc:1398] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:38.599167 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session e139f2784fa5493983f786b885d7c33d-a63bd458e2e94636a3a64ddc59ad8196 received from {username='slave'} at 127.0.99.134:59511
I20240503 01:59:38.599260 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session e139f2784fa5493983f786b885d7c33d-a63bd458e2e94636a3a64ddc59ad8196 on tablet a63bd458e2e94636a3a64ddc59ad8196 with peer e139f2784fa5493983f786b885d7c33d
I20240503 01:59:38.613212 3894 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.613485 3894 tablet_bootstrap.cc:492] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap complete.
I20240503 01:59:38.613644 3894 ts_tablet_manager.cc:1367] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240503 01:59:38.614112 3894 raft_consensus.cc:348] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:38.614221 3894 raft_consensus.cc:729] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07d11a6a41ef47b888dae403ffbf6456, State: Initialized, Role: LEARNER
I20240503 01:59:38.614320 3894 consensus_queue.cc:260] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:38.614602 3894 ts_tablet_manager.cc:1398] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:38.615164 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session 07d11a6a41ef47b888dae403ffbf6456-3fd5de36fcad46a0bf6aca04394e46c7 received from {username='slave'} at 127.0.99.135:35807
I20240503 01:59:38.615259 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session 07d11a6a41ef47b888dae403ffbf6456-3fd5de36fcad46a0bf6aca04394e46c7 on tablet 3fd5de36fcad46a0bf6aca04394e46c7 with peer 07d11a6a41ef47b888dae403ffbf6456
W20240503 01:59:38.647034 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 01:59:38.819926 3914 tablet_copy_client.cc:523] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.821323 3914 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Bootstrap starting.
I20240503 01:59:38.840679 3914 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.840914 3914 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Bootstrap complete.
I20240503 01:59:38.841048 3914 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240503 01:59:38.841365 3914 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:38.841467 3914 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Initialized, Role: LEARNER
I20240503 01:59:38.841548 3914 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:38.841866 3914 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:38.842355 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session 091deff9eeda46b4ab93b05a3f466847-83a04a16e66b463b98976c490155139c received from {username='slave'} at 127.0.99.133:45561
I20240503 01:59:38.842442 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session 091deff9eeda46b4ab93b05a3f466847-83a04a16e66b463b98976c490155139c on tablet 83a04a16e66b463b98976c490155139c with peer 091deff9eeda46b4ab93b05a3f466847
W20240503 01:59:38.878799 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 42cb1c96b35a4b98a1edfa4ccd99602d: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:38.889715 2131 raft_consensus.cc:1203] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.906666 1999 raft_consensus.cc:1203] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 01:59:39.005542 2262 raft_consensus.cc:1203] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 01:59:39.043581 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 01:59:39.351213 4086 raft_consensus.cc:1050] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER 091deff9eeda46b4ab93b05a3f466847 to VOTER
I20240503 01:59:39.351686 4086 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:39.353035 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.353390 4086 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.353471 975 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.353874 4086 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.354517 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:39.354954 4086 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:39.356351 3939 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.357192 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.358318 1999 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.358827 440 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.359522 3988 raft_consensus.cc:1050] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER e139f2784fa5493983f786b885d7c33d to VOTER
I20240503 01:59:39.359858 975 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.359956 3988 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:39.361941 844 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.362049 975 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.362349 2131 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.362740 4060 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.363134 4060 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.363504 3988 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.365206 3988 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.365722 975 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.368484 845 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.368537 433 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 4 to 5, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.368691 844 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:39.370038 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.370002 2131 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.370420 4086 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:39.371060 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 01:59:39.371423 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 01:59:39.371939 4086 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:39.372830 4086 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.374001 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.374495 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83a04a16e66b463b98976c490155139c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.374677 1999 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.375588 713 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 01:59:39.376865 2131 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.376978 437 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 reported cstate change: config changed from index 9 to 10, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:39.377189 975 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.377257 3988 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.377619 3988 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.379276 3988 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.379707 975 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.379952 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a04a16e66b463b98976c490155139c with delete_type TABLET_DATA_TOMBSTONED (TS 648ac6ef81784cf3a916a2ed46655b3f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43320
I20240503 01:59:39.379921 2131 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 01:59:39.381088 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a63bd458e2e94636a3a64ddc59ad8196 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.381785 432 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 5 to 6, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.382747 4109 tablet_replica.cc:330] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: stopping tablet replica
I20240503 01:59:39.382923 4109 raft_consensus.cc:2229] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.383129 4109 raft_consensus.cc:2258] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.384132 824 tablet_service.cc:1514] Processing DeleteTablet for tablet a63bd458e2e94636a3a64ddc59ad8196 with delete_type TABLET_DATA_TOMBSTONED (TS c0801789471740eda4246b1ab4904e10 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57380
I20240503 01:59:39.384546 4110 tablet_replica.cc:330] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: stopping tablet replica
I20240503 01:59:39.384673 4110 raft_consensus.cc:2229] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.384825 4110 raft_consensus.cc:2258] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.385685 4110 ts_tablet_manager.cc:1875] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.385767 4109 ts_tablet_manager.cc:1875] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.387609 4109 ts_tablet_manager.cc:1888] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 01:59:39.387686 4110 ts_tablet_manager.cc:1888] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.387746 4109 log.cc:1196] T 83a04a16e66b463b98976c490155139c P 648ac6ef81784cf3a916a2ed46655b3f: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/83a04a16e66b463b98976c490155139c
I20240503 01:59:39.387887 4110 log.cc:1196] T a63bd458e2e94636a3a64ddc59ad8196 P c0801789471740eda4246b1ab4904e10: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a63bd458e2e94636a3a64ddc59ad8196
I20240503 01:59:39.388271 428 catalog_manager.cc:4830] TS 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053): tablet 83a04a16e66b463b98976c490155139c (table rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]) successfully deleted
I20240503 01:59:39.388438 428 catalog_manager.cc:4830] TS c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797): tablet a63bd458e2e94636a3a64ddc59ad8196 (table rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]) successfully deleted
I20240503 01:59:39.447280 4088 raft_consensus.cc:1050] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 to VOTER
I20240503 01:59:39.447719 4088 consensus_queue.cc:237] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:39.449059 2262 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.449152 975 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.449290 713 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:39.449452 4112 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.449694 4086 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.449868 3939 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.451536 4086 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.452212 975 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.452373 713 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.454521 437 catalog_manager.cc:5484] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:39.455274 2262 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.457702 845 consensus_queue.cc:237] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:39.458891 975 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.459043 2262 raft_consensus.cc:1261] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 01:59:39.459260 4088 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.459786 4088 consensus_queue.cc:1035] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:39.461022 4086 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.461563 2262 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.462091 975 raft_consensus.cc:2932] T 3fd5de36fcad46a0bf6aca04394e46c7 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.463292 440 catalog_manager.cc:5484] T 3fd5de36fcad46a0bf6aca04394e46c7 P 07d11a6a41ef47b888dae403ffbf6456 reported cstate change: config changed from index 5 to 6, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:39.463443 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fd5de36fcad46a0bf6aca04394e46c7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:39.465544 694 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd5de36fcad46a0bf6aca04394e46c7 with delete_type TABLET_DATA_TOMBSTONED (TS efd7ce56deab4305a002577febb80532 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49820
I20240503 01:59:39.466066 4116 tablet_replica.cc:330] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: stopping tablet replica
I20240503 01:59:39.466383 4116 raft_consensus.cc:2229] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.466604 4116 raft_consensus.cc:2258] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.467563 4116 ts_tablet_manager.cc:1875] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:39.469319 4116 ts_tablet_manager.cc:1888] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 01:59:39.469401 4116 log.cc:1196] T 3fd5de36fcad46a0bf6aca04394e46c7 P efd7ce56deab4305a002577febb80532: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fd5de36fcad46a0bf6aca04394e46c7
I20240503 01:59:39.469894 425 catalog_manager.cc:4830] TS efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): tablet 3fd5de36fcad46a0bf6aca04394e46c7 (table rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]) successfully deleted
I20240503 01:59:39.586593 713 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:39.588235 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.588570 845 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:39.588701 4100 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.589077 4100 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:39.590879 4060 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:39.591403 845 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:39.592118 1999 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
W20240503 01:59:39.593899 650 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 -> Peer 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161): Couldn't send request to peer 07d11a6a41ef47b888dae403ffbf6456. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42cb1c96b35a4b98a1edfa4ccd99602d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:39.594619 432 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 7 to 8, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
W20240503 01:59:39.792213 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 01:59:39.933099 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 01:59:40.029815 4156 ts_tablet_manager.cc:902] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:40.030272 4156 tablet_copy_client.cc:308] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:40.033578 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer 07d11a6a41ef47b888dae403ffbf6456 ({username='slave'} at 127.0.99.135:43067)
I20240503 01:59:40.033689 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer 07d11a6a41ef47b888dae403ffbf6456 at {username='slave'} at 127.0.99.135:43067: session id = 07d11a6a41ef47b888dae403ffbf6456-42cb1c96b35a4b98a1edfa4ccd99602d
I20240503 01:59:40.034538 734 tablet_copy_source_session.cc:205] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:40.035367 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.037345 4156 tablet_copy_client.cc:783] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:40.037503 4156 tablet_copy_client.cc:647] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:40.038429 4160 tablet_copy_client.cc:1002] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:40.375245 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0c46bbf4f1b435993ce1057b2b4de9e: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:40.409917 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 01:59:40.921973 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 01:59:41.412916 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 01:59:41.525988 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 01:59:41.538750 4156 tablet_copy_client.cc:523] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:41.540159 4156 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap starting.
I20240503 01:59:41.561468 4156 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:41.561741 4156 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap complete.
I20240503 01:59:41.561884 4156 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.001s
I20240503 01:59:41.562232 4156 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:41.562335 4156 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07d11a6a41ef47b888dae403ffbf6456, State: Initialized, Role: LEARNER
I20240503 01:59:41.562412 4156 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:41.562674 4156 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:41.563270 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session 07d11a6a41ef47b888dae403ffbf6456-42cb1c96b35a4b98a1edfa4ccd99602d received from {username='slave'} at 127.0.99.135:43067
I20240503 01:59:41.563367 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session 07d11a6a41ef47b888dae403ffbf6456-42cb1c96b35a4b98a1edfa4ccd99602d on tablet 42cb1c96b35a4b98a1edfa4ccd99602d with peer 07d11a6a41ef47b888dae403ffbf6456
W20240503 01:59:41.729022 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 01:59:41.906554 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 01:59:42.085078 2262 raft_consensus.cc:1203] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 01:59:42.236222 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 01:59:42.365228 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 01:59:42.399269 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 01:59:42.399978 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba2b3e448315456789bfa5953e80c7ce: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:42.530274 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 01:59:42.608678 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 01:59:42.698273 4245 raft_consensus.cc:1050] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 to VOTER
I20240503 01:59:42.698743 4245 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:42.700183 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:42.700296 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:42.700296 845 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:42.700623 4247 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:42.700842 4245 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:42.701038 4188 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:42.702826 4245 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:42.703238 1999 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:42.703466 2262 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:42.705332 437 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 8 to 9, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:42.705376 845 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:42.907444 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "efd7ce56deab4305a002577febb80532"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58550
I20240503 01:59:42.907641 713 raft_consensus.cc:595] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Received request to transfer leadership
W20240503 01:59:42.923743 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 01:59:43.118170 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "efd7ce56deab4305a002577febb80532"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58572
I20240503 01:59:43.118324 713 raft_consensus.cc:595] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:43.119082 4247 raft_consensus.cc:979] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: : Instructing follower c0801789471740eda4246b1ab4904e10 to start an election
I20240503 01:59:43.119405 4244 raft_consensus.cc:1067] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Signalling peer c0801789471740eda4246b1ab4904e10 to start an election
I20240503 01:59:43.119956 845 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
from {username='slave'} at 127.0.99.130:55621
I20240503 01:59:43.120092 845 raft_consensus.cc:482] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:43.120143 845 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:43.120889 845 raft_consensus.cc:504] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:43.121405 845 leader_election.cc:288] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 election: Requested vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:43.121603 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:43.121724 713 raft_consensus.cc:3032] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:43.121763 713 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:43.121799 1999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "091deff9eeda46b4ab93b05a3f466847"
I20240503 01:59:43.121979 1999 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:43.121943 713 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:43.122000 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 01:59:43.122107 2262 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:43.122138 713 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:43.122759 1999 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 2.
I20240503 01:59:43.123055 713 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 2.
I20240503 01:59:43.123057 2262 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 2.
I20240503 01:59:43.123327 778 leader_election.cc:302] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 091deff9eeda46b4ab93b05a3f466847, c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:43.123559 4088 raft_consensus.cc:2781] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:43.123715 4088 raft_consensus.cc:686] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:43.123876 4088 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:43.125627 433 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 1 to 2, leader changed from efd7ce56deab4305a002577febb80532 (127.0.99.130) to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:43.532421 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 01:59:43.539012 713 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:43.539471 4088 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:43.562773 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:43.563208 4088 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:43.566321 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:43.566916 4088 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 01:59:43.748862 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 01:59:44.061218 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 01:59:44.162300 845 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:44.163352 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:44.163410 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:44.163771 4315 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:44.164008 4314 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:44.165676 4314 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:44.166023 1999 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:44.167027 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:44.167470 2262 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:44.168295 432 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 9 to 11, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:44.171562 694 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with delete_type TABLET_DATA_TOMBSTONED (TS efd7ce56deab4305a002577febb80532 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49820
I20240503 01:59:44.171926 4323 tablet_replica.cc:330] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: stopping tablet replica
I20240503 01:59:44.172084 4323 raft_consensus.cc:2229] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.172207 4323 raft_consensus.cc:2258] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.172789 4323 ts_tablet_manager.cc:1875] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.174437 4323 ts_tablet_manager.cc:1888] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 01:59:44.174500 4323 log.cc:1196] T 42cb1c96b35a4b98a1edfa4ccd99602d P efd7ce56deab4305a002577febb80532: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/42cb1c96b35a4b98a1edfa4ccd99602d
I20240503 01:59:44.174922 425 catalog_manager.cc:4830] TS efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): tablet 42cb1c96b35a4b98a1edfa4ccd99602d (table rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]) successfully deleted
I20240503 01:59:44.363794 713 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
W20240503 01:59:44.365347 650 consensus_peers.cc:485] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 -> Peer 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161): Couldn't send request to peer 07d11a6a41ef47b888dae403ffbf6456. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba2b3e448315456789bfa5953e80c7ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:44.365432 845 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:44.365610 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:44.365787 4337 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:44.366178 4336 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:44.367542 4337 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:44.368077 845 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:44.368635 2131 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:44.370982 440 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 8 to 9, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:44.469645 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 01:59:44.941998 4357 ts_tablet_manager.cc:902] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:44.942441 4357 tablet_copy_client.cc:308] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:44.942940 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet ba2b3e448315456789bfa5953e80c7ce from peer 07d11a6a41ef47b888dae403ffbf6456 ({username='slave'} at 127.0.99.135:43067)
I20240503 01:59:44.943054 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet ba2b3e448315456789bfa5953e80c7ce from peer 07d11a6a41ef47b888dae403ffbf6456 at {username='slave'} at 127.0.99.135:43067: session id = 07d11a6a41ef47b888dae403ffbf6456-ba2b3e448315456789bfa5953e80c7ce
I20240503 01:59:44.943882 734 tablet_copy_source_session.cc:205] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:44.944712 4357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2b3e448315456789bfa5953e80c7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.946570 4357 tablet_copy_client.cc:783] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:44.946731 4357 tablet_copy_client.cc:647] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:44.947708 4361 tablet_copy_client.cc:1002] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:45.038393 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 01:59:45.122488 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 01:59:45.475080 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 01:59:45.695036 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8018cb49863d4736ae5f79f472f63bb8: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:46.273910 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 01:59:46.366955 4088 consensus_queue.cc:579] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.179s)
W20240503 01:59:46.368327 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:46.434381 4108 consensus_queue.cc:579] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.312s)
W20240503 01:59:46.435846 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:46.448001 4357 tablet_copy_client.cc:523] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.449522 4357 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap starting.
I20240503 01:59:46.470338 4357 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.470639 4357 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap complete.
I20240503 01:59:46.470799 4357 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240503 01:59:46.471197 4357 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:46.471310 4357 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 07d11a6a41ef47b888dae403ffbf6456, State: Initialized, Role: LEARNER
I20240503 01:59:46.471393 4357 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:46.471720 4357 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:46.472318 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session 07d11a6a41ef47b888dae403ffbf6456-ba2b3e448315456789bfa5953e80c7ce received from {username='slave'} at 127.0.99.135:43067
I20240503 01:59:46.472419 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session 07d11a6a41ef47b888dae403ffbf6456-ba2b3e448315456789bfa5953e80c7ce on tablet ba2b3e448315456789bfa5953e80c7ce with peer 07d11a6a41ef47b888dae403ffbf6456
I20240503 01:59:46.560351 2262 raft_consensus.cc:1203] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 01:59:46.609731 4417 consensus_queue.cc:579] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.415s)
W20240503 01:59:46.703683 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 01:59:46.917462 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 01:59:47.045266 4397 raft_consensus.cc:1050] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 to VOTER
I20240503 01:59:47.045763 4397 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:47.047238 2262 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:47.047283 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:47.047400 845 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:47.047688 4434 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:47.047890 4433 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:47.048133 4397 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:47.050330 4397 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:47.050793 2131 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:47.050823 2262 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:47.053052 845 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:47.053331 433 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 reported cstate change: config changed from index 9 to 10, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:47.062352 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce"
dest_uuid: "efd7ce56deab4305a002577febb80532"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43868
I20240503 01:59:47.062454 713 raft_consensus.cc:595] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Received request to transfer leadership
I20240503 01:59:47.272640 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce"
dest_uuid: "efd7ce56deab4305a002577febb80532"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43898
I20240503 01:59:47.272787 713 raft_consensus.cc:595] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Received request to transfer leadership
W20240503 01:59:47.303555 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 01:59:47.304179 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:47.454730 4397 raft_consensus.cc:979] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: : Instructing follower c0801789471740eda4246b1ab4904e10 to start an election
I20240503 01:59:47.454839 4397 raft_consensus.cc:1067] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Signalling peer c0801789471740eda4246b1ab4904e10 to start an election
I20240503 01:59:47.455374 845 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
from {username='slave'} at 127.0.99.130:55621
I20240503 01:59:47.455528 845 raft_consensus.cc:482] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:47.455602 845 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:47.456568 845 raft_consensus.cc:504] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:47.457114 845 leader_election.cc:288] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 3 election: Requested vote from peers efd7ce56deab4305a002577febb80532 (127.0.99.130:46503), e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:47.457324 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "efd7ce56deab4305a002577febb80532"
I20240503 01:59:47.457438 713 raft_consensus.cc:3032] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 01:59:47.457479 713 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: efd7ce56deab4305a002577febb80532, State: Running, Role: LEADER
I20240503 01:59:47.457494 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 01:59:47.457602 713 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:47.457630 2262 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:47.457731 713 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:47.458474 2262 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 3.
I20240503 01:59:47.458645 713 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 3.
I20240503 01:59:47.458906 778 leader_election.cc:302] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 07d11a6a41ef47b888dae403ffbf6456, c0801789471740eda4246b1ab4904e10, efd7ce56deab4305a002577febb80532; no voters:
I20240503 01:59:47.459060 4430 raft_consensus.cc:2781] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 01:59:47.459137 4430 raft_consensus.cc:686] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 3 LEADER]: Becoming Leader. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:47.459276 4430 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:47.460984 437 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 reported cstate change: term changed from 2 to 3, leader changed from efd7ce56deab4305a002577febb80532 (127.0.99.130) to c0801789471740eda4246b1ab4904e10 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:47.461364 2131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2b3e448315456789bfa5953e80c7ce" candidate_uuid: "c0801789471740eda4246b1ab4904e10" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "e139f2784fa5493983f786b885d7c33d"
I20240503 01:59:47.461508 2131 raft_consensus.cc:3037] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:47.462405 2131 raft_consensus.cc:2454] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0801789471740eda4246b1ab4904e10 in term 3.
W20240503 01:59:47.601897 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:47.630693 4397 consensus_queue.cc:579] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.064s)
W20240503 01:59:47.631357 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 01:59:47.856755 713 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 3 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 01:59:47.857189 4430 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:47.862711 4088 consensus_queue.cc:579] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.344s)
I20240503 01:59:47.891952 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 01:59:47.892386 4475 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:47.896385 2262 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 01:59:47.896911 4430 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:47.992843 4417 consensus_queue.cc:579] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.207s)
W20240503 01:59:47.994364 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:48.145956 4487 consensus_queue.cc:579] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Leader has been unable to successfully communicate with peer 0a141082e3c24126a1542e2c9a3aad0e for more than 30 seconds (30.319s)
I20240503 01:59:48.336335 845 consensus_queue.cc:237] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:48.337549 2262 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 01:59:48.337720 2131 raft_consensus.cc:1261] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 01:59:48.337937 4475 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:48.338120 4088 consensus_queue.cc:1035] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:48.339627 4475 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:48.340046 2131 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:48.341293 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba2b3e448315456789bfa5953e80c7ce with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:48.341683 2262 raft_consensus.cc:2932] T ba2b3e448315456789bfa5953e80c7ce P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:48.341811 440 catalog_manager.cc:5484] T ba2b3e448315456789bfa5953e80c7ce P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 10 to 12, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:48.344511 694 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2b3e448315456789bfa5953e80c7ce with delete_type TABLET_DATA_TOMBSTONED (TS efd7ce56deab4305a002577febb80532 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49820
I20240503 01:59:48.344856 4503 tablet_replica.cc:330] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: stopping tablet replica
I20240503 01:59:48.345018 4503 raft_consensus.cc:2229] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.345218 4503 raft_consensus.cc:2258] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.345896 4503 ts_tablet_manager.cc:1875] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.347985 4503 ts_tablet_manager.cc:1888] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 01:59:48.348045 4503 log.cc:1196] T ba2b3e448315456789bfa5953e80c7ce P efd7ce56deab4305a002577febb80532: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba2b3e448315456789bfa5953e80c7ce
I20240503 01:59:48.348510 425 catalog_manager.cc:4830] TS efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): tablet ba2b3e448315456789bfa5953e80c7ce (table rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]) successfully deleted
I20240503 01:59:48.577515 1980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:48.578610 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:48.579393 2373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:48.579895 2111 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:48.581324 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:48.581637 824 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:48.591449 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 01:59:48.615334 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 01:59:48.616214 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0c46bbf4f1b435993ce1057b2b4de9e: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
I20240503 01:59:48.633646 713 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:48.635102 975 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.635092 2131 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 01:59:48.635488 4558 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.635869 4558 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 01:59:48.637826 4559 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:48.638214 2131 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
W20240503 01:59:48.639976 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 01:59:48.640482 432 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 6 to 7, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
W20240503 01:59:48.640795 648 consensus_peers.cc:485] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 -> Peer b6db6ee3749340658049b2301a938993 (127.0.99.136:34021): Couldn't send request to peer b6db6ee3749340658049b2301a938993. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a63bd458e2e94636a3a64ddc59ad8196. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.640981 975 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:48.648687 2131 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:48.650074 713 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 01:59:48.650120 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 01:59:48.650473 4570 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:48.650725 4569 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:48.652367 4569 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:48.652818 713 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:48.654448 2393 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
W20240503 01:59:48.654930 2066 consensus_peers.cc:485] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d -> Peer 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505): Couldn't send request to peer 091deff9eeda46b4ab93b05a3f466847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0c46bbf4f1b435993ce1057b2b4de9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.655325 432 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 reported cstate change: config changed from index 11 to 12, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:48.663388 845 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:48.664724 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 01:59:48.664745 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 01:59:48.664988 781 consensus_peers.cc:485] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 -> Peer 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161): Couldn't send request to peer 07d11a6a41ef47b888dae403ffbf6456. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83a04a16e66b463b98976c490155139c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:48.665180 4088 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:48.665416 4475 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:48.667083 4430 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:48.667565 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:48.668782 1999 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } } }
I20240503 01:59:48.669589 437 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 10 to 11, NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:48.870699 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 01:59:48.898856 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 01:59:49.018532 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 01:59:49.021776 4614 ts_tablet_manager.cc:902] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Initiating tablet copy from peer efd7ce56deab4305a002577febb80532 (127.0.99.130:46503)
I20240503 01:59:49.022271 4614 tablet_copy_client.cc:308] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46503
I20240503 01:59:49.025519 734 tablet_copy_service.cc:140] P efd7ce56deab4305a002577febb80532: Received BeginTabletCopySession request for tablet a63bd458e2e94636a3a64ddc59ad8196 from peer b6db6ee3749340658049b2301a938993 ({username='slave'} at 127.0.99.136:59739)
I20240503 01:59:49.025626 734 tablet_copy_service.cc:161] P efd7ce56deab4305a002577febb80532: Beginning new tablet copy session on tablet a63bd458e2e94636a3a64ddc59ad8196 from peer b6db6ee3749340658049b2301a938993 at {username='slave'} at 127.0.99.136:59739: session id = b6db6ee3749340658049b2301a938993-a63bd458e2e94636a3a64ddc59ad8196
I20240503 01:59:49.026366 734 tablet_copy_source_session.cc:205] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:49.027185 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63bd458e2e94636a3a64ddc59ad8196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.029146 4614 tablet_copy_client.cc:783] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:49.029287 4614 tablet_copy_client.cc:647] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:49.030249 4618 tablet_copy_client.cc:1002] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:49.098132 4620 ts_tablet_manager.cc:902] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Initiating tablet copy from peer e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873)
I20240503 01:59:49.098688 4620 tablet_copy_client.cc:308] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:34873
I20240503 01:59:49.102277 2151 tablet_copy_service.cc:140] P e139f2784fa5493983f786b885d7c33d: Received BeginTabletCopySession request for tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer 091deff9eeda46b4ab93b05a3f466847 ({username='slave'} at 127.0.99.133:50203)
I20240503 01:59:49.102396 2151 tablet_copy_service.cc:161] P e139f2784fa5493983f786b885d7c33d: Beginning new tablet copy session on tablet b0c46bbf4f1b435993ce1057b2b4de9e from peer 091deff9eeda46b4ab93b05a3f466847 at {username='slave'} at 127.0.99.133:50203: session id = 091deff9eeda46b4ab93b05a3f466847-b0c46bbf4f1b435993ce1057b2b4de9e
I20240503 01:59:49.103456 2151 tablet_copy_source_session.cc:205] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:49.104537 4620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0c46bbf4f1b435993ce1057b2b4de9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.106248 4620 tablet_copy_client.cc:783] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:49.106400 4620 tablet_copy_client.cc:647] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:49.107574 4624 tablet_copy_client.cc:1002] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:49.207973 4641 ts_tablet_manager.cc:902] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:49.208424 4641 tablet_copy_client.cc:308] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:49.208923 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 83a04a16e66b463b98976c490155139c from peer 07d11a6a41ef47b888dae403ffbf6456 ({username='slave'} at 127.0.99.135:35807)
I20240503 01:59:49.209041 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 83a04a16e66b463b98976c490155139c from peer 07d11a6a41ef47b888dae403ffbf6456 at {username='slave'} at 127.0.99.135:35807: session id = 07d11a6a41ef47b888dae403ffbf6456-83a04a16e66b463b98976c490155139c
I20240503 01:59:49.209774 865 tablet_copy_source_session.cc:205] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:49.210626 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a04a16e66b463b98976c490155139c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.212391 4641 tablet_copy_client.cc:783] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:49.212543 4641 tablet_copy_client.cc:647] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:49.213515 4645 tablet_copy_client.cc:1002] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:49.980343 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 01:59:50.276908 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 01:59:50.365679 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 01:59:50.530539 4614 tablet_copy_client.cc:523] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.531826 4614 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Bootstrap starting.
I20240503 01:59:50.549436 4614 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.549703 4614 tablet_bootstrap.cc:492] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Bootstrap complete.
I20240503 01:59:50.549845 4614 ts_tablet_manager.cc:1367] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20240503 01:59:50.550208 4614 raft_consensus.cc:348] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:50.550312 4614 raft_consensus.cc:729] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6db6ee3749340658049b2301a938993, State: Initialized, Role: LEARNER
I20240503 01:59:50.550390 4614 consensus_queue.cc:260] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:50.550683 4614 ts_tablet_manager.cc:1398] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:50.551322 734 tablet_copy_service.cc:342] P efd7ce56deab4305a002577febb80532: Request end of tablet copy session b6db6ee3749340658049b2301a938993-a63bd458e2e94636a3a64ddc59ad8196 received from {username='slave'} at 127.0.99.136:59739
I20240503 01:59:50.551404 734 tablet_copy_service.cc:433] P efd7ce56deab4305a002577febb80532: ending tablet copy session b6db6ee3749340658049b2301a938993-a63bd458e2e94636a3a64ddc59ad8196 on tablet a63bd458e2e94636a3a64ddc59ad8196 with peer b6db6ee3749340658049b2301a938993
I20240503 01:59:50.607892 4620 tablet_copy_client.cc:523] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.609259 4620 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Bootstrap starting.
I20240503 01:59:50.628909 4620 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.629237 4620 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Bootstrap complete.
I20240503 01:59:50.629422 4620 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240503 01:59:50.629851 4620 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:50.629961 4620 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Initialized, Role: LEARNER
I20240503 01:59:50.630048 4620 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:50.630394 4620 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:50.631441 2151 tablet_copy_service.cc:342] P e139f2784fa5493983f786b885d7c33d: Request end of tablet copy session 091deff9eeda46b4ab93b05a3f466847-b0c46bbf4f1b435993ce1057b2b4de9e received from {username='slave'} at 127.0.99.133:50203
I20240503 01:59:50.631551 2151 tablet_copy_service.cc:433] P e139f2784fa5493983f786b885d7c33d: ending tablet copy session 091deff9eeda46b4ab93b05a3f466847-b0c46bbf4f1b435993ce1057b2b4de9e on tablet b0c46bbf4f1b435993ce1057b2b4de9e with peer 091deff9eeda46b4ab93b05a3f466847
I20240503 01:59:50.680780 1999 raft_consensus.cc:1203] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 01:59:50.713829 4641 tablet_copy_client.cc:523] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:50.715171 4641 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap starting.
I20240503 01:59:50.736878 4641 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:50.737149 4641 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Bootstrap complete.
I20240503 01:59:50.737320 4641 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 01:59:50.737720 4641 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:50.737840 4641 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07d11a6a41ef47b888dae403ffbf6456, State: Initialized, Role: LEARNER
I20240503 01:59:50.737934 4641 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: true } }
I20240503 01:59:50.738246 4641 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:50.738786 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session 07d11a6a41ef47b888dae403ffbf6456-83a04a16e66b463b98976c490155139c received from {username='slave'} at 127.0.99.135:35807
I20240503 01:59:50.738864 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session 07d11a6a41ef47b888dae403ffbf6456-83a04a16e66b463b98976c490155139c on tablet 83a04a16e66b463b98976c490155139c with peer 07d11a6a41ef47b888dae403ffbf6456
I20240503 01:59:51.142949 2393 raft_consensus.cc:1203] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 01:59:51.196741 2262 raft_consensus.cc:1203] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 01:59:51.230628 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 01:59:51.290140 4818 raft_consensus.cc:1050] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d: attempting to promote NON_VOTER 091deff9eeda46b4ab93b05a3f466847 to VOTER
I20240503 01:59:51.290620 4818 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:51.292237 1999 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:51.292237 713 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:51.292245 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:51.292572 4817 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:51.292750 4721 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:51.292901 4818 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:51.294894 4721 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.295867 713 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.296044 1999 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.298058 433 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 12 to 13, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:51.298547 2393 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.302441 2131 consensus_queue.cc:237] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:51.303524 1999 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 01:59:51.303567 2393 raft_consensus.cc:1261] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 01:59:51.303893 4824 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:51.304054 4817 consensus_queue.cc:1035] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:51.305440 4817 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.305918 2393 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.306650 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0c46bbf4f1b435993ce1057b2b4de9e with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:51.307582 440 catalog_manager.cc:5484] T b0c46bbf4f1b435993ce1057b2b4de9e P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 13 to 14, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:51.308665 1999 raft_consensus.cc:2932] T b0c46bbf4f1b435993ce1057b2b4de9e P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:51.309849 694 tablet_service.cc:1514] Processing DeleteTablet for tablet b0c46bbf4f1b435993ce1057b2b4de9e with delete_type TABLET_DATA_TOMBSTONED (TS efd7ce56deab4305a002577febb80532 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49820
I20240503 01:59:51.310194 4834 tablet_replica.cc:330] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: stopping tablet replica
I20240503 01:59:51.310402 4834 raft_consensus.cc:2229] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.310638 4834 raft_consensus.cc:2258] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.311266 4834 ts_tablet_manager.cc:1875] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:51.313174 4834 ts_tablet_manager.cc:1888] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 01:59:51.313249 4834 log.cc:1196] T b0c46bbf4f1b435993ce1057b2b4de9e P efd7ce56deab4305a002577febb80532: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0c46bbf4f1b435993ce1057b2b4de9e
I20240503 01:59:51.313750 425 catalog_manager.cc:4830] TS efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): tablet b0c46bbf4f1b435993ce1057b2b4de9e (table rebalance_test_table_2 [id=55e37a37eca6411fa49776607bbfb059]) successfully deleted
W20240503 01:59:51.383441 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 01:59:51.407819 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 01:59:51.502646 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 01:59:51.523020 975 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:51.524492 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:51.524703 713 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 01:59:51.524946 4858 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.525254 4487 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.526860 4858 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:51.527338 713 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
I20240503 01:59:51.527369 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } } }
W20240503 01:59:51.529603 911 consensus_peers.cc:485] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505): Couldn't send request to peer 091deff9eeda46b4ab93b05a3f466847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4266aabe93664eb889ebf7cca050e3c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:51.529845 437 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 7 to 8, NON_VOTER 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 01:59:51.747943 4880 raft_consensus.cc:1050] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532: attempting to promote NON_VOTER b6db6ee3749340658049b2301a938993 to VOTER
I20240503 01:59:51.748425 4880 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:51.750059 975 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.750198 2393 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 LEARNER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.750207 2131 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 01:59:51.750483 4879 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.750737 4880 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.750917 4664 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 01:59:51.752624 4879 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.753000 2131 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.753739 975 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.755295 2393 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.755571 440 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 7 to 8, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.759735 713 consensus_queue.cc:237] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:51.760982 2131 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:51.761049 2393 raft_consensus.cc:1261] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer efd7ce56deab4305a002577febb80532: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 01:59:51.761336 4664 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:51.762053 4664 consensus_queue.cc:1035] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:51.763051 4880 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P efd7ce56deab4305a002577febb80532 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.763505 2131 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.764416 2393 raft_consensus.cc:2932] T a63bd458e2e94636a3a64ddc59ad8196 P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.765997 437 catalog_manager.cc:5484] T a63bd458e2e94636a3a64ddc59ad8196 P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 8 to 9, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "efd7ce56deab4305a002577febb80532" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:51.766937 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a63bd458e2e94636a3a64ddc59ad8196 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:51.768752 956 tablet_service.cc:1514] Processing DeleteTablet for tablet a63bd458e2e94636a3a64ddc59ad8196 with delete_type TABLET_DATA_TOMBSTONED (TS 648ac6ef81784cf3a916a2ed46655b3f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43320
I20240503 01:59:51.769323 4895 tablet_replica.cc:330] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: stopping tablet replica
I20240503 01:59:51.769469 4895 raft_consensus.cc:2229] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.769672 4895 raft_consensus.cc:2258] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.770838 4895 ts_tablet_manager.cc:1875] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:51.772773 4895 ts_tablet_manager.cc:1888] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 01:59:51.772835 4895 log.cc:1196] T a63bd458e2e94636a3a64ddc59ad8196 P 648ac6ef81784cf3a916a2ed46655b3f: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a63bd458e2e94636a3a64ddc59ad8196
I20240503 01:59:51.773347 428 catalog_manager.cc:4830] TS 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053): tablet a63bd458e2e94636a3a64ddc59ad8196 (table rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]) successfully deleted
I20240503 01:59:51.785135 4773 raft_consensus.cc:1050] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER 07d11a6a41ef47b888dae403ffbf6456 to VOTER
I20240503 01:59:51.785598 4773 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:51.786988 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:51.787045 2262 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:51.787119 2393 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 01:59:51.787436 4914 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:51.787633 4911 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:51.787927 4773 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:51.789826 4773 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:51.790307 2393 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:51.790578 2262 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:51.791867 1999 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:51.792456 432 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 11 to 12, 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 01:59:51.845707 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83a04a16e66b463b98976c490155139c: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:51.977810 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 01:59:52.003083 845 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83a04a16e66b463b98976c490155139c"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33452
I20240503 01:59:52.003216 845 raft_consensus.cc:595] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:52.048735 4935 ts_tablet_manager.cc:902] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Initiating tablet copy from peer 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:52.049274 4935 tablet_copy_client.cc:308] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34053
I20240503 01:59:52.052459 996 tablet_copy_service.cc:140] P 648ac6ef81784cf3a916a2ed46655b3f: Received BeginTabletCopySession request for tablet 4266aabe93664eb889ebf7cca050e3c2 from peer 091deff9eeda46b4ab93b05a3f466847 ({username='slave'} at 127.0.99.133:60191)
I20240503 01:59:52.052551 996 tablet_copy_service.cc:161] P 648ac6ef81784cf3a916a2ed46655b3f: Beginning new tablet copy session on tablet 4266aabe93664eb889ebf7cca050e3c2 from peer 091deff9eeda46b4ab93b05a3f466847 at {username='slave'} at 127.0.99.133:60191: session id = 091deff9eeda46b4ab93b05a3f466847-4266aabe93664eb889ebf7cca050e3c2
I20240503 01:59:52.053287 996 tablet_copy_source_session.cc:205] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:52.054086 4935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.055783 4935 tablet_copy_client.cc:783] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:52.055943 4935 tablet_copy_client.cc:647] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:52.056878 4939 tablet_copy_client.cc:1002] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:52.193706 4911 raft_consensus.cc:979] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: : Instructing follower b6db6ee3749340658049b2301a938993 to start an election
I20240503 01:59:52.193845 4911 raft_consensus.cc:1067] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Signalling peer b6db6ee3749340658049b2301a938993 to start an election
I20240503 01:59:52.194291 2393 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83a04a16e66b463b98976c490155139c"
dest_uuid: "b6db6ee3749340658049b2301a938993"
from {username='slave'} at 127.0.99.131:34009
I20240503 01:59:52.194478 2393 raft_consensus.cc:482] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:52.194542 2393 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:52.195434 2393 raft_consensus.cc:504] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:52.196580 2393 leader_election.cc:288] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [CANDIDATE]: Term 2 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161)
I20240503 01:59:52.197297 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "b6db6ee3749340658049b2301a938993" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:52.197422 845 raft_consensus.cc:3032] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:52.197463 845 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:52.197607 845 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:52.197778 845 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:52.198767 845 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6db6ee3749340658049b2301a938993 in term 2.
I20240503 01:59:52.201718 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "b6db6ee3749340658049b2301a938993" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 01:59:52.201850 2262 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:52.202297 1999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a04a16e66b463b98976c490155139c" candidate_uuid: "b6db6ee3749340658049b2301a938993" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "091deff9eeda46b4ab93b05a3f466847"
I20240503 01:59:52.202430 1999 raft_consensus.cc:3037] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:52.202827 2262 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6db6ee3749340658049b2301a938993 in term 2.
I20240503 01:59:52.203146 2329 leader_election.cc:302] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 07d11a6a41ef47b888dae403ffbf6456, b6db6ee3749340658049b2301a938993, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:52.203409 1999 raft_consensus.cc:2454] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6db6ee3749340658049b2301a938993 in term 2.
I20240503 01:59:52.203400 4806 raft_consensus.cc:2781] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:52.203596 4806 raft_consensus.cc:686] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 2 LEADER]: Becoming Leader. State: Replica: b6db6ee3749340658049b2301a938993, State: Running, Role: LEADER
I20240503 01:59:52.203923 4806 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:52.206449 433 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 reported cstate change: term changed from 1 to 2, leader changed from c0801789471740eda4246b1ab4904e10 (127.0.99.131) to b6db6ee3749340658049b2301a938993 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "b6db6ee3749340658049b2301a938993" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:52.456729 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 01:59:52.524924 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 01:59:52.691321 845 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Refusing update from remote peer b6db6ee3749340658049b2301a938993: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:52.692050 4806 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 01:59:52.728808 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 01:59:52.801354 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer b6db6ee3749340658049b2301a938993: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:52.801841 4806 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:52.805037 2262 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer b6db6ee3749340658049b2301a938993: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:52.805423 4983 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 01:59:52.806962 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 01:59:53.043864 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 01:59:53.290863 2393 consensus_queue.cc:237] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } }
I20240503 01:59:53.292021 2262 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer b6db6ee3749340658049b2301a938993: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 01:59:53.292081 1999 raft_consensus.cc:1261] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer b6db6ee3749340658049b2301a938993: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 01:59:53.292403 4983 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:53.292640 4987 consensus_queue.cc:1035] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:53.294181 4983 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:53.294823 2262 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:53.295895 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83a04a16e66b463b98976c490155139c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:53.296305 1999 raft_consensus.cc:2932] T 83a04a16e66b463b98976c490155139c P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } }
I20240503 01:59:53.296996 433 catalog_manager.cc:5484] T 83a04a16e66b463b98976c490155139c P b6db6ee3749340658049b2301a938993 reported cstate change: config changed from index 12 to 14, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "b6db6ee3749340658049b2301a938993" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:53.300093 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a04a16e66b463b98976c490155139c with delete_type TABLET_DATA_TOMBSTONED (TS c0801789471740eda4246b1ab4904e10 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:57380
I20240503 01:59:53.300474 5027 tablet_replica.cc:330] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: stopping tablet replica
I20240503 01:59:53.300617 5027 raft_consensus.cc:2229] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:53.300791 5027 raft_consensus.cc:2258] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:53.301353 5027 ts_tablet_manager.cc:1875] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:53.303069 5027 ts_tablet_manager.cc:1888] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 01:59:53.303133 5027 log.cc:1196] T 83a04a16e66b463b98976c490155139c P c0801789471740eda4246b1ab4904e10: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/83a04a16e66b463b98976c490155139c
I20240503 01:59:53.303597 428 catalog_manager.cc:4830] TS c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797): tablet 83a04a16e66b463b98976c490155139c (table rebalance_test_table_4 [id=34ae4959f6f341bdbd2f3a530fdede8e]) successfully deleted
I20240503 01:59:53.498200 845 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:53.499840 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 01:59:53.500075 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 01:59:53.500123 779 consensus_peers.cc:485] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 -> Peer b6db6ee3749340658049b2301a938993 (127.0.99.136:34021): Couldn't send request to peer b6db6ee3749340658049b2301a938993. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42cb1c96b35a4b98a1edfa4ccd99602d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.500504 5046 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:53.500696 5045 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 01:59:53.502362 5045 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:53.502981 2262 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:53.503058 1999 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:53.505585 440 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 reported cstate change: config changed from index 11 to 12, NON_VOTER b6db6ee3749340658049b2301a938993 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } } }
I20240503 01:59:53.557212 4935 tablet_copy_client.cc:523] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.558876 4935 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap starting.
I20240503 01:59:53.578080 4935 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.578388 4935 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Bootstrap complete.
I20240503 01:59:53.578547 4935 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240503 01:59:53.578941 4935 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:53.579056 4935 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Initialized, Role: LEARNER
I20240503 01:59:53.579130 4935 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: true } }
I20240503 01:59:53.579376 4935 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:53.579967 996 tablet_copy_service.cc:342] P 648ac6ef81784cf3a916a2ed46655b3f: Request end of tablet copy session 091deff9eeda46b4ab93b05a3f466847-4266aabe93664eb889ebf7cca050e3c2 received from {username='slave'} at 127.0.99.133:60191
I20240503 01:59:53.580049 996 tablet_copy_service.cc:433] P 648ac6ef81784cf3a916a2ed46655b3f: ending tablet copy session 091deff9eeda46b4ab93b05a3f466847-4266aabe93664eb889ebf7cca050e3c2 on tablet 4266aabe93664eb889ebf7cca050e3c2 with peer 091deff9eeda46b4ab93b05a3f466847
W20240503 01:59:53.594097 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 01:59:53.732111 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 01:59:53.884838 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 01:59:54.030769 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 01:59:54.107156 5083 ts_tablet_manager.cc:902] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Initiating tablet copy from peer c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797)
I20240503 01:59:54.107746 5083 tablet_copy_client.cc:308] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:38797
I20240503 01:59:54.108371 865 tablet_copy_service.cc:140] P c0801789471740eda4246b1ab4904e10: Received BeginTabletCopySession request for tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer b6db6ee3749340658049b2301a938993 ({username='slave'} at 127.0.99.136:50053)
I20240503 01:59:54.108484 865 tablet_copy_service.cc:161] P c0801789471740eda4246b1ab4904e10: Beginning new tablet copy session on tablet 42cb1c96b35a4b98a1edfa4ccd99602d from peer b6db6ee3749340658049b2301a938993 at {username='slave'} at 127.0.99.136:50053: session id = b6db6ee3749340658049b2301a938993-42cb1c96b35a4b98a1edfa4ccd99602d
I20240503 01:59:54.109282 865 tablet_copy_source_session.cc:205] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:54.110124 5083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42cb1c96b35a4b98a1edfa4ccd99602d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.111922 5083 tablet_copy_client.cc:783] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:54.112066 5083 tablet_copy_client.cc:647] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:54.113094 5087 tablet_copy_client.cc:1002] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: tablet copy: Injecting latency into file download: 1500
I20240503 01:59:54.119230 1999 raft_consensus.cc:1203] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 01:59:54.645838 5125 raft_consensus.cc:1050] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: attempting to promote NON_VOTER 091deff9eeda46b4ab93b05a3f466847 to VOTER
I20240503 01:59:54.646314 5125 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:54.647665 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 LEARNER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:54.647819 713 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:54.648093 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 01:59:54.648195 5124 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:54.648531 5125 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:54.648933 5127 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 01:59:54.650580 5124 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.651044 1999 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.652532 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.653194 713 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.653206 433 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 8 to 9, 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:54.657358 975 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } }
I20240503 01:59:54.658499 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:54.658499 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 01:59:54.658871 5125 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:54.659036 5124 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 01:59:54.660460 4487 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.660853 1999 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.662197 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4266aabe93664eb889ebf7cca050e3c2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:54.662662 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.662861 440 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 reported cstate change: config changed from index 9 to 10, VOTER efd7ce56deab4305a002577febb80532 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } }
I20240503 01:59:54.665486 694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 with delete_type TABLET_DATA_TOMBSTONED (TS efd7ce56deab4305a002577febb80532 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49820
I20240503 01:59:54.665807 5137 tablet_replica.cc:330] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: stopping tablet replica
I20240503 01:59:54.665951 5137 raft_consensus.cc:2229] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.666152 5137 raft_consensus.cc:2258] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.666850 5137 ts_tablet_manager.cc:1875] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:54.668601 5137 ts_tablet_manager.cc:1888] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 01:59:54.668669 5137 log.cc:1196] T 4266aabe93664eb889ebf7cca050e3c2 P efd7ce56deab4305a002577febb80532: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4266aabe93664eb889ebf7cca050e3c2
I20240503 01:59:54.669138 425 catalog_manager.cc:4830] TS efd7ce56deab4305a002577febb80532 (127.0.99.130:46503): tablet 4266aabe93664eb889ebf7cca050e3c2 (table rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]) successfully deleted
W20240503 01:59:54.990628 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 01:59:55.290515 427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 01:59:55.291188 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 42cb1c96b35a4b98a1edfa4ccd99602d: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:55.496748 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 01:59:55.613368 5083 tablet_copy_client.cc:523] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:55.614668 5083 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Bootstrap starting.
I20240503 01:59:55.635993 5083 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:55.636271 5083 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Bootstrap complete.
I20240503 01:59:55.636408 5083 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 01:59:55.636777 5083 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:55.636881 5083 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b6db6ee3749340658049b2301a938993, State: Initialized, Role: LEARNER
I20240503 01:59:55.636961 5083 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 01:59:55.637203 5083 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:55.637725 865 tablet_copy_service.cc:342] P c0801789471740eda4246b1ab4904e10: Request end of tablet copy session b6db6ee3749340658049b2301a938993-42cb1c96b35a4b98a1edfa4ccd99602d received from {username='slave'} at 127.0.99.136:50053
I20240503 01:59:55.637809 865 tablet_copy_service.cc:433] P c0801789471740eda4246b1ab4904e10: ending tablet copy session b6db6ee3749340658049b2301a938993-42cb1c96b35a4b98a1edfa4ccd99602d on tablet 42cb1c96b35a4b98a1edfa4ccd99602d with peer b6db6ee3749340658049b2301a938993
I20240503 01:59:55.697526 2393 raft_consensus.cc:1203] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240503 01:59:55.761440 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 01:59:56.161954 5195 raft_consensus.cc:1050] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: attempting to promote NON_VOTER b6db6ee3749340658049b2301a938993 to VOTER
I20240503 01:59:56.162385 5195 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:56.163698 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:56.163698 2393 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 LEARNER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:56.163709 1999 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer c0801789471740eda4246b1ab4904e10: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 01:59:56.164171 5195 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:56.164376 5162 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:56.164566 5197 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 01:59:56.166335 5162 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:56.167109 2262 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:56.167912 2393 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:56.168798 437 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 reported cstate change: config changed from index 12 to 13, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c0801789471740eda4246b1ab4904e10" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 01:59:56.168802 1999 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b6db6ee3749340658049b2301a938993 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:56.247722 845 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42038
I20240503 01:59:56.247900 845 raft_consensus.cc:595] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Received request to transfer leadership
W20240503 01:59:56.250947 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 01:59:56.292312 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 01:59:56.413863 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 01:59:56.458647 845 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "c0801789471740eda4246b1ab4904e10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42052
I20240503 01:59:56.458781 845 raft_consensus.cc:595] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Received request to transfer leadership
W20240503 01:59:56.460410 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 01:59:56.572600 5197 raft_consensus.cc:979] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: : Instructing follower 091deff9eeda46b4ab93b05a3f466847 to start an election
I20240503 01:59:56.572746 5197 raft_consensus.cc:1067] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Signalling peer 091deff9eeda46b4ab93b05a3f466847 to start an election
I20240503 01:59:56.573220 1999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d"
dest_uuid: "091deff9eeda46b4ab93b05a3f466847"
from {username='slave'} at 127.0.99.131:55905
I20240503 01:59:56.573424 1999 raft_consensus.cc:482] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:56.573540 1999 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:56.574417 1999 raft_consensus.cc:504] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:56.575429 1999 leader_election.cc:288] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [CANDIDATE]: Term 3 election: Requested vote from peers c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161), b6db6ee3749340658049b2301a938993 (127.0.99.136:34021)
I20240503 01:59:56.576009 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "091deff9eeda46b4ab93b05a3f466847" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "c0801789471740eda4246b1ab4904e10"
I20240503 01:59:56.576141 845 raft_consensus.cc:3032] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 01:59:56.576181 845 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0801789471740eda4246b1ab4904e10, State: Running, Role: LEADER
I20240503 01:59:56.576337 845 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:56.576565 845 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:56.577484 845 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091deff9eeda46b4ab93b05a3f466847 in term 3.
I20240503 01:59:56.579931 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "091deff9eeda46b4ab93b05a3f466847" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 01:59:56.579957 2393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cb1c96b35a4b98a1edfa4ccd99602d" candidate_uuid: "091deff9eeda46b4ab93b05a3f466847" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "b6db6ee3749340658049b2301a938993"
I20240503 01:59:56.580056 2262 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:56.580067 2393 raft_consensus.cc:3037] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 2 FOLLOWER]: Advancing to term 3
I20240503 01:59:56.580803 2262 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091deff9eeda46b4ab93b05a3f466847 in term 3.
I20240503 01:59:56.580808 2393 raft_consensus.cc:2454] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 091deff9eeda46b4ab93b05a3f466847 in term 3.
I20240503 01:59:56.581126 1934 leader_election.cc:302] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 091deff9eeda46b4ab93b05a3f466847, b6db6ee3749340658049b2301a938993, c0801789471740eda4246b1ab4904e10; no voters:
I20240503 01:59:56.581351 5203 raft_consensus.cc:2781] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 01:59:56.581578 5203 raft_consensus.cc:686] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 3 LEADER]: Becoming Leader. State: Replica: 091deff9eeda46b4ab93b05a3f466847, State: Running, Role: LEADER
I20240503 01:59:56.581923 5203 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:56.584358 440 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 reported cstate change: term changed from 2 to 3, leader changed from c0801789471740eda4246b1ab4904e10 (127.0.99.131) to 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133). New cstate: current_term: 3 leader_uuid: "091deff9eeda46b4ab93b05a3f466847" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 01:59:56.985262 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 01:59:57.031826 845 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Refusing update from remote peer 091deff9eeda46b4ab93b05a3f466847: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 01:59:57.032294 5203 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:57.115855 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer 091deff9eeda46b4ab93b05a3f466847: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 01:59:57.116299 5203 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 01:59:57.119900 2393 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 3 FOLLOWER]: Refusing update from remote peer 091deff9eeda46b4ab93b05a3f466847: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 01:59:57.120684 5258 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 01:59:57.407639 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 01:59:57.464151 780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 01:59:57.654717 1999 consensus_queue.cc:237] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } }
I20240503 01:59:57.655918 2393 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 3 FOLLOWER]: Refusing update from remote peer 091deff9eeda46b4ab93b05a3f466847: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 01:59:57.655931 2262 raft_consensus.cc:1261] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer 091deff9eeda46b4ab93b05a3f466847: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 01:59:57.656288 5270 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 01:59:57.656529 5271 consensus_queue.cc:1035] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 01:59:57.658089 5270 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 091deff9eeda46b4ab93b05a3f466847 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:57.658591 2393 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P b6db6ee3749340658049b2301a938993 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:57.658607 2262 raft_consensus.cc:2932] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:57.660449 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 01:59:57.661075 432 catalog_manager.cc:5484] T 42cb1c96b35a4b98a1edfa4ccd99602d P 07d11a6a41ef47b888dae403ffbf6456 reported cstate change: config changed from index 13 to 15, VOTER c0801789471740eda4246b1ab4904e10 (127.0.99.131) evicted. New cstate: current_term: 3 leader_uuid: "091deff9eeda46b4ab93b05a3f466847" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: false } } }
I20240503 01:59:57.663743 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 42cb1c96b35a4b98a1edfa4ccd99602d with delete_type TABLET_DATA_TOMBSTONED (TS c0801789471740eda4246b1ab4904e10 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:57380
I20240503 01:59:57.664333 5278 tablet_replica.cc:330] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: stopping tablet replica
I20240503 01:59:57.664480 5278 raft_consensus.cc:2229] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.664714 5278 raft_consensus.cc:2258] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.665351 5278 ts_tablet_manager.cc:1875] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:57.667186 5278 ts_tablet_manager.cc:1888] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240503 01:59:57.667253 5278 log.cc:1196] T 42cb1c96b35a4b98a1edfa4ccd99602d P c0801789471740eda4246b1ab4904e10: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/42cb1c96b35a4b98a1edfa4ccd99602d
I20240503 01:59:57.667707 428 catalog_manager.cc:4830] TS c0801789471740eda4246b1ab4904e10 (127.0.99.131:38797): tablet 42cb1c96b35a4b98a1edfa4ccd99602d (table rebalance_test_table_0 [id=af5056fd77b640e8b3b426c641fe2a57]) successfully deleted
I20240503 01:59:57.759495 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:57.760046 2373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.760165 1980 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.765084 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:57.766742 956 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:57.776573 2111 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:57.793444 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:57.819295 975 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:57.820667 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 01:59:57.820686 2262 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240503 01:59:57.820943 911 consensus_peers.cc:485] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer e139f2784fa5493983f786b885d7c33d (127.0.99.134:34873): Couldn't send request to peer e139f2784fa5493983f786b885d7c33d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4266aabe93664eb889ebf7cca050e3c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:57.821103 5330 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:57.821275 4487 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 01:59:57.823185 5330 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:57.823451 2262 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:57.823558 1999 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
I20240503 01:59:57.825870 433 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 reported cstate change: config changed from index 10 to 11, NON_VOTER e139f2784fa5493983f786b885d7c33d (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } } }
W20240503 01:59:57.914325 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 01:59:57.982756 649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 01:59:58.357836 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 01:59:58.424027 5348 ts_tablet_manager.cc:902] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Initiating tablet copy from peer 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053)
I20240503 01:59:58.424579 5348 tablet_copy_client.cc:308] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:34053
I20240503 01:59:58.425056 996 tablet_copy_service.cc:140] P 648ac6ef81784cf3a916a2ed46655b3f: Received BeginTabletCopySession request for tablet 4266aabe93664eb889ebf7cca050e3c2 from peer e139f2784fa5493983f786b885d7c33d ({username='slave'} at 127.0.99.134:48149)
I20240503 01:59:58.425187 996 tablet_copy_service.cc:161] P 648ac6ef81784cf3a916a2ed46655b3f: Beginning new tablet copy session on tablet 4266aabe93664eb889ebf7cca050e3c2 from peer e139f2784fa5493983f786b885d7c33d at {username='slave'} at 127.0.99.134:48149: session id = e139f2784fa5493983f786b885d7c33d-4266aabe93664eb889ebf7cca050e3c2
I20240503 01:59:58.425921 996 tablet_copy_source_session.cc:205] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:58.426759 5348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4266aabe93664eb889ebf7cca050e3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.428496 5348 tablet_copy_client.cc:783] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:58.428642 5348 tablet_copy_client.cc:647] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:58.429600 5352 tablet_copy_client.cc:1002] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: tablet copy: Injecting latency into file download: 1500
W20240503 01:59:58.570045 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 01:59:58.588374 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 01:59:58.816311 427 catalog_manager.cc:4573] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ba2b3e448315456789bfa5953e80c7ce: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111)
W20240503 01:59:59.038859 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 01:59:59.179968 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 01:59:59.929906 5348 tablet_copy_client.cc:523] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:59.931478 5348 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Bootstrap starting.
I20240503 01:59:59.955214 5348 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:59.955487 5348 tablet_bootstrap.cc:492] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Bootstrap complete.
I20240503 01:59:59.955652 5348 ts_tablet_manager.cc:1367] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.001s
I20240503 01:59:59.956085 5348 raft_consensus.cc:348] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:59.956199 5348 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Initialized, Role: LEARNER
I20240503 01:59:59.956297 5348 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: true } }
I20240503 01:59:59.956549 5348 ts_tablet_manager.cc:1398] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:59.957262 996 tablet_copy_service.cc:342] P 648ac6ef81784cf3a916a2ed46655b3f: Request end of tablet copy session e139f2784fa5493983f786b885d7c33d-4266aabe93664eb889ebf7cca050e3c2 received from {username='slave'} at 127.0.99.134:48149
I20240503 01:59:59.957350 996 tablet_copy_service.cc:433] P 648ac6ef81784cf3a916a2ed46655b3f: ending tablet copy session e139f2784fa5493983f786b885d7c33d-4266aabe93664eb889ebf7cca050e3c2 on tablet 4266aabe93664eb889ebf7cca050e3c2 with peer e139f2784fa5493983f786b885d7c33d
I20240503 02:00:00.032112 2131 raft_consensus.cc:1203] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240503 02:00:00.076938 911 consensus_peers.cc:485] T d7ba19da3d1f4a4182fcc967a8ca6314 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:00:00.226344 780 consensus_peers.cc:485] T 4f75e2ba8c494fc5bde411b10b687860 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:00:00.426673 5418 raft_consensus.cc:1050] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: attempting to promote NON_VOTER e139f2784fa5493983f786b885d7c33d to VOTER
I20240503 02:00:00.427083 5418 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 02:00:00.428514 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:00.428601 2131 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 LEARNER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:00.428946 2261 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Refusing update from remote peer 648ac6ef81784cf3a916a2ed46655b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:00:00.429064 5418 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:00.429261 5388 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:00.429406 5417 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:00:00.431509 5388 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:00.432026 2261 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:00.432574 1999 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:00.434170 2131 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:00.434290 436 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f reported cstate change: config changed from index 11 to 12, e139f2784fa5493983f786b885d7c33d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "648ac6ef81784cf3a916a2ed46655b3f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:00.517706 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45274
I20240503 02:00:00.517849 975 raft_consensus.cc:595] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Received request to transfer leadership
I20240503 02:00:00.728793 975 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45304
I20240503 02:00:00.728921 975 raft_consensus.cc:595] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Received request to transfer leadership
W20240503 02:00:00.781636 911 consensus_peers.cc:485] T e6835fae0d5049f1a7e31fa51fd58f7a P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:00:00.872005 5418 raft_consensus.cc:979] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: : Instructing follower e139f2784fa5493983f786b885d7c33d to start an election
I20240503 02:00:00.872126 5418 raft_consensus.cc:1067] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Signalling peer e139f2784fa5493983f786b885d7c33d to start an election
I20240503 02:00:00.872529 2131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2"
dest_uuid: "e139f2784fa5493983f786b885d7c33d"
from {username='slave'} at 127.0.99.132:47105
I20240503 02:00:00.872666 2131 raft_consensus.cc:482] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:00.872731 2131 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:00.873528 2131 raft_consensus.cc:504] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 02:00:00.874004 2131 leader_election.cc:288] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [CANDIDATE]: Term 3 election: Requested vote from peers 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053), 07d11a6a41ef47b888dae403ffbf6456 (127.0.99.135:37161), 091deff9eeda46b4ab93b05a3f466847 (127.0.99.133:36505)
I20240503 02:00:00.874282 975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "648ac6ef81784cf3a916a2ed46655b3f"
I20240503 02:00:00.874356 1999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "091deff9eeda46b4ab93b05a3f466847"
I20240503 02:00:00.874402 975 raft_consensus.cc:3032] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:00.874442 975 raft_consensus.cc:729] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 648ac6ef81784cf3a916a2ed46655b3f, State: Running, Role: LEADER
I20240503 02:00:00.874475 1999 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:00.874588 975 consensus_queue.cc:260] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 02:00:00.874783 975 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:00.875459 1999 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 3.
I20240503 02:00:00.875742 975 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 3.
I20240503 02:00:00.876056 2067 leader_election.cc:302] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 091deff9eeda46b4ab93b05a3f466847, 648ac6ef81784cf3a916a2ed46655b3f, e139f2784fa5493983f786b885d7c33d; no voters:
I20240503 02:00:00.876216 5394 raft_consensus.cc:2781] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:00.876371 5394 raft_consensus.cc:686] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 3 LEADER]: Becoming Leader. State: Replica: e139f2784fa5493983f786b885d7c33d, State: Running, Role: LEADER
I20240503 02:00:00.876523 5394 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 02:00:00.877935 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4266aabe93664eb889ebf7cca050e3c2" candidate_uuid: "e139f2784fa5493983f786b885d7c33d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "07d11a6a41ef47b888dae403ffbf6456"
I20240503 02:00:00.878053 2261 raft_consensus.cc:3037] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:00.878257 436 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d reported cstate change: term changed from 2 to 3, leader changed from 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) to e139f2784fa5493983f786b885d7c33d (127.0.99.134). New cstate: current_term: 3 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:00.878823 2261 raft_consensus.cc:2454] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e139f2784fa5493983f786b885d7c33d in term 3.
W20240503 02:00:01.133589 911 consensus_peers.cc:485] T bae517688d34428e9a4983829376002b P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:01.223217 911 consensus_peers.cc:485] T 66c65746b14f430ca238ee3a9f5cf008 P 648ac6ef81784cf3a916a2ed46655b3f -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:00:01.349426 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:00:01.349814 5394 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:01.481833 2261 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:00:01.482277 5394 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:01.484908 975 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 3 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:00:01.485697 5474 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:00:01.487613 2131 consensus_queue.cc:237] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } }
I20240503 02:00:01.488731 2261 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:00:01.488742 1999 raft_consensus.cc:1261] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Refusing update from remote peer e139f2784fa5493983f786b885d7c33d: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:00:01.489091 5472 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:00:01.489238 5474 consensus_queue.cc:1035] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:00:01.490559 5394 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:01.491063 1999 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 091deff9eeda46b4ab93b05a3f466847 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:01.491099 2261 raft_consensus.cc:2932] T 4266aabe93664eb889ebf7cca050e3c2 P 07d11a6a41ef47b888dae403ffbf6456 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } } }
I20240503 02:00:01.492641 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4266aabe93664eb889ebf7cca050e3c2 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:00:01.493244 433 catalog_manager.cc:5484] T 4266aabe93664eb889ebf7cca050e3c2 P e139f2784fa5493983f786b885d7c33d reported cstate change: config changed from index 12 to 14, VOTER 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132) evicted. New cstate: current_term: 3 leader_uuid: "e139f2784fa5493983f786b885d7c33d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "07d11a6a41ef47b888dae403ffbf6456" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091deff9eeda46b4ab93b05a3f466847" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36505 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e139f2784fa5493983f786b885d7c33d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 34873 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:00:01.495890 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 4266aabe93664eb889ebf7cca050e3c2 with delete_type TABLET_DATA_TOMBSTONED (TS 648ac6ef81784cf3a916a2ed46655b3f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:43320
I20240503 02:00:01.496212 5482 tablet_replica.cc:330] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: stopping tablet replica
I20240503 02:00:01.496366 5482 raft_consensus.cc:2229] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.496481 5482 raft_consensus.cc:2258] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.497071 5482 ts_tablet_manager.cc:1875] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:01.498844 5482 ts_tablet_manager.cc:1888] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240503 02:00:01.498919 5482 log.cc:1196] T 4266aabe93664eb889ebf7cca050e3c2 P 648ac6ef81784cf3a916a2ed46655b3f: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4266aabe93664eb889ebf7cca050e3c2
I20240503 02:00:01.499936 428 catalog_manager.cc:4830] TS 648ac6ef81784cf3a916a2ed46655b3f (127.0.99.132:34053): tablet 4266aabe93664eb889ebf7cca050e3c2 (table rebalance_test_table_3 [id=e485566e81e348eb8ad19e2605a43df6]) successfully deleted
W20240503 02:00:01.592665 780 consensus_peers.cc:485] T 3bd70004ec0a4244ad89c59b0fab01c7 P c0801789471740eda4246b1ab4904e10 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:00:01.610546 649 consensus_peers.cc:485] T 481dd137b628482b9f1333aceb89ae39 P efd7ce56deab4305a002577febb80532 -> Peer 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): Couldn't send request to peer 0a141082e3c24126a1542e2c9a3aad0e. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:39079: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:00:01.610910 2373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:01.610917 956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:01.616581 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:01.617952 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:01.619982 1980 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:01.628803 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:01.630393 2111 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:01.718813 2373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:01.722712 956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:01.728493 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:01.731330 1980 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:01.732337 2111 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:01.732492 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:01.737637 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:01.785132 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:39079
--local_ip_for_outbound_sockets=127.0.99.129
--tserver_master_addrs=127.0.99.190:35171
--webserver_port=35049
--webserver_interface=127.0.99.129
--builtin_ntp_servers=127.0.99.148:40617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:00:01.892146 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:01.892357 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:01.892432 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:01.895432 5562 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:00:01.895504 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:01.895601 5562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:01.899230 5562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:39079
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=35049
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:35171
--never_fsync=true
--heap_profile_path=/tmp/kudu.5562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:01.900179 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:01.901269 5562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:01.909997 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:01.910775 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:01.911588 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:01.912397 5562 server_base.cc:1034] running on GCE node
I20240503 02:00:01.912727 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:01.913283 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:01.914419 5562 hybrid_clock.cc:648] HybridClock initialized: now 1714701601914400 us; error 34 us; skew 500 ppm
I20240503 02:00:01.916285 5562 webserver.cc:460] Webserver started at http://127.0.99.129:35049/ using document root <none> and password file <none>
I20240503 02:00:01.916812 5562 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:01.916906 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:01.920536 5562 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:01.923102 5576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:01.924505 5562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:01.924618 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0a141082e3c24126a1542e2c9a3aad0e"
format_stamp: "Formatted at 2024-05-03 01:59:06 on dist-test-slave-fvr5"
I20240503 02:00:01.925068 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:01.940289 5562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:01.940940 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:01.941156 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:01.941857 5562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:01.943665 5583 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240503 02:00:01.954948 5562 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240503 02:00:01.955021 5562 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20240503 02:00:01.955081 5562 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240503 02:00:01.957314 5583 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:01.963245 5562 ts_tablet_manager.cc:590] Registered 12 tablets
I20240503 02:00:01.963301 5562 ts_tablet_manager.cc:569] Time spent register tablets: real 0.008s user 0.001s sys 0.004s
I20240503 02:00:01.975526 5562 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:39079
I20240503 02:00:01.975561 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:39079 every 8 connection(s)
I20240503 02:00:01.975943 5583 log.cc:824] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:01.976701 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:01.981516 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 5562
I20240503 02:00:01.985227 5692 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35171
I20240503 02:00:01.985427 5692 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:01.985956 5692 heartbeater.cc:511] Master 127.0.99.190:35171 requested a full tablet report, sending...
I20240503 02:00:01.987710 433 ts_manager.cc:194] Re-registered known tserver with Master: 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079)
I20240503 02:00:01.989900 433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:39941
I20240503 02:00:02.008397 5583 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.008844 5583 tablet_bootstrap.cc:492] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.010038 5583 ts_tablet_manager.cc:1367] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.053s user 0.042s sys 0.008s
I20240503 02:00:02.012487 5583 raft_consensus.cc:348] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 02:00:02.012898 5583 raft_consensus.cc:729] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.013394 5583 consensus_queue.cc:260] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } attrs { replace: true } } peers { permanent_uuid: "b6db6ee3749340658049b2301a938993" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 34021 } attrs { promote: true } }
I20240503 02:00:02.014165 5692 heartbeater.cc:503] Master 127.0.99.190:35171 was elected leader, sending a full tablet report...
I20240503 02:00:02.014241 5583 ts_tablet_manager.cc:1398] T 83a04a16e66b463b98976c490155139c P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 02:00:02.014492 5583 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.046989 5583 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.047456 5583 tablet_bootstrap.cc:492] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.048556 5583 ts_tablet_manager.cc:1367] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20240503 02:00:02.049093 5583 raft_consensus.cc:348] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 02:00:02.049239 5583 raft_consensus.cc:729] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.049391 5583 consensus_queue.cc:260] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 02:00:02.049661 5583 ts_tablet_manager.cc:1398] T d7ba19da3d1f4a4182fcc967a8ca6314 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.049825 5583 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.080763 5583 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.081182 5583 tablet_bootstrap.cc:492] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.082335 5583 ts_tablet_manager.cc:1367] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.011s
I20240503 02:00:02.082787 5583 raft_consensus.cc:348] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 02:00:02.082916 5583 raft_consensus.cc:729] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.083055 5583 consensus_queue.cc:260] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 02:00:02.083302 5583 ts_tablet_manager.cc:1398] T 66c65746b14f430ca238ee3a9f5cf008 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.083451 5583 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.089447 5625 tablet_service.cc:1514] Processing DeleteTablet for tablet 8018cb49863d4736ae5f79f472f63bb8 with delete_type TABLET_DATA_TOMBSTONED (TS 0a141082e3c24126a1542e2c9a3aad0e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49722
W20240503 02:00:02.090282 427 catalog_manager.cc:4810] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): delete failed for tablet 8018cb49863d4736ae5f79f472f63bb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8018cb49863d4736ae5f79f472f63bb8 already in progress: opening tablet
I20240503 02:00:02.110306 5583 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.110664 5583 tablet_bootstrap.cc:492] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.111620 5583 ts_tablet_manager.cc:1367] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.004s
I20240503 02:00:02.112313 5583 raft_consensus.cc:348] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.112507 5583 raft_consensus.cc:729] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.112679 5583 consensus_queue.cc:260] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.112990 5583 ts_tablet_manager.cc:1398] T 3bd70004ec0a4244ad89c59b0fab01c7 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.113206 5583 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.146270 5583 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.146869 5583 tablet_bootstrap.cc:492] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.148011 5583 ts_tablet_manager.cc:1367] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.014s
I20240503 02:00:02.148393 5583 raft_consensus.cc:348] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.148494 5583 raft_consensus.cc:729] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.148641 5583 consensus_queue.cc:260] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.148895 5583 ts_tablet_manager.cc:1398] T ba2b3e448315456789bfa5953e80c7ce P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:02.149082 5583 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.177124 5583 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.177465 5583 tablet_bootstrap.cc:492] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.178894 5583 ts_tablet_manager.cc:1367] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.005s
I20240503 02:00:02.179349 5583 raft_consensus.cc:348] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.179473 5583 raft_consensus.cc:729] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.179597 5583 consensus_queue.cc:260] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.179937 5583 ts_tablet_manager.cc:1398] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:02.180155 5583 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.213342 5583 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.213750 5583 tablet_bootstrap.cc:492] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.214910 5583 ts_tablet_manager.cc:1367] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.008s
I20240503 02:00:02.215420 5583 raft_consensus.cc:348] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.215551 5583 raft_consensus.cc:729] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.215731 5583 consensus_queue.cc:260] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.216132 5583 ts_tablet_manager.cc:1398] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:02.216274 5583 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.218648 5625 tablet_service.cc:1514] Processing DeleteTablet for tablet 8018cb49863d4736ae5f79f472f63bb8 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 7)) from {username='slave'} at 127.0.0.1:49722
I20240503 02:00:02.219076 5701 tablet_replica.cc:330] stopping tablet replica
I20240503 02:00:02.219326 5701 raft_consensus.cc:2229] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.219543 5701 raft_consensus.cc:2258] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.220678 5701 ts_tablet_manager.cc:1875] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:02.223701 5701 ts_tablet_manager.cc:1888] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:00:02.223805 5701 log.cc:1196] T 8018cb49863d4736ae5f79f472f63bb8 P 0a141082e3c24126a1542e2c9a3aad0e: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8018cb49863d4736ae5f79f472f63bb8
I20240503 02:00:02.224349 427 catalog_manager.cc:4830] TS 0a141082e3c24126a1542e2c9a3aad0e (127.0.99.129:39079): tablet 8018cb49863d4736ae5f79f472f63bb8 (table rebalance_test_table_1 [id=bf15305d536443ba90b6c95aec2fa7c6]) successfully deleted
I20240503 02:00:02.239742 1980 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:02.239804 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:02.242066 5625 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:02.246311 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:02.251170 5583 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.251449 5583 tablet_bootstrap.cc:492] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.252434 5583 ts_tablet_manager.cc:1367] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.008s
I20240503 02:00:02.252903 5583 raft_consensus.cc:348] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 02:00:02.253024 5583 raft_consensus.cc:729] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.253159 5583 consensus_queue.cc:260] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } }
I20240503 02:00:02.253434 5583 ts_tablet_manager.cc:1398] T 4f75e2ba8c494fc5bde411b10b687860 P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.253635 5583 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.262029 2111 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:02.282111 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:02.285764 2373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:02.286044 956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:02.299582 5583 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.300247 5583 tablet_bootstrap.cc:492] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.301375 5583 ts_tablet_manager.cc:1367] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.048s user 0.032s sys 0.004s
I20240503 02:00:02.301847 5583 raft_consensus.cc:348] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.301985 5583 raft_consensus.cc:729] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.302147 5583 consensus_queue.cc:260] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } }
I20240503 02:00:02.304270 5583 ts_tablet_manager.cc:1398] T 42cb1c96b35a4b98a1edfa4ccd99602d P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:02.306021 5583 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.337898 5583 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.338255 5583 tablet_bootstrap.cc:492] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.339247 5583 ts_tablet_manager.cc:1367] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20240503 02:00:02.339704 5583 raft_consensus.cc:348] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.339843 5583 raft_consensus.cc:729] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.339982 5583 consensus_queue.cc:260] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.340256 5583 ts_tablet_manager.cc:1398] T e6835fae0d5049f1a7e31fa51fd58f7a P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:02.340394 5583 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.366345 5583 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.366689 5583 tablet_bootstrap.cc:492] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.367779 5583 ts_tablet_manager.cc:1367] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.000s
I20240503 02:00:02.368145 5583 raft_consensus.cc:348] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 02:00:02.368243 5583 raft_consensus.cc:729] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.368381 5583 consensus_queue.cc:260] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } } peers { permanent_uuid: "efd7ce56deab4305a002577febb80532" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46503 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } }
I20240503 02:00:02.368649 5583 ts_tablet_manager.cc:1398] T b0c46bbf4f1b435993ce1057b2b4de9e P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.368804 5583 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap starting.
I20240503 02:00:02.399309 5583 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.399708 5583 tablet_bootstrap.cc:492] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Bootstrap complete.
I20240503 02:00:02.401341 5583 ts_tablet_manager.cc:1367] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.004s
I20240503 02:00:02.401799 5583 raft_consensus.cc:348] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.401907 5583 raft_consensus.cc:729] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a141082e3c24126a1542e2c9a3aad0e, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.402025 5583 consensus_queue.cc:260] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0801789471740eda4246b1ab4904e10" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38797 } } peers { permanent_uuid: "648ac6ef81784cf3a916a2ed46655b3f" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34053 } } peers { permanent_uuid: "0a141082e3c24126a1542e2c9a3aad0e" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 39079 } }
I20240503 02:00:02.402307 5583 ts_tablet_manager.cc:1398] T bae517688d34428e9a4983829376002b P 0a141082e3c24126a1542e2c9a3aad0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:02.405054 1980 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:02.405577 2242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:02.417546 5645 raft_consensus.cc:3037] T 481dd137b628482b9f1333aceb89ae39 P 0a141082e3c24126a1542e2c9a3aad0e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:02.420882 956 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:02.425335 5625 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:02.431631 2373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:02.431838 694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:02.434185 824 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:02.443873 2111 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:02.487035 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 5562
I20240503 02:00:02.493685 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 632
I20240503 02:00:02.501892 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 763
I20240503 02:00:02.507203 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 894
I20240503 02:00:02.513705 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 1918
I20240503 02:00:02.519924 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 2049
I20240503 02:00:02.525727 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 2180
I20240503 02:00:02.531538 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 2311
I20240503 02:00:02.537158 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 410
2024-05-03T02:00:02Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (56815 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (56815 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 02:00:02.573364 398 test_util.cc:268] Using random seed: 334154799
I20240503 02:00:02.574160 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:02.574203 398 ts_itest-base.cc:113] --------------
I20240503 02:00:02.574224 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:00:02.574242 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:00:02.574270 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:02Z Disabled control of system clock
I20240503 02:00:02.582062 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39653
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:39653
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:00:02.688211 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:02.688446 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:02.688513 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:02.691437 5807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:02.691509 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:02.691550 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:02.691587 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:02.691629 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:02.695196 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:39653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:02.696185 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:02.697291 5807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:02.705093 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:02.705825 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:02.706585 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:02.707329 5807 server_base.cc:1034] running on GCE node
I20240503 02:00:02.707851 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:02.708739 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:02.709932 5807 hybrid_clock.cc:648] HybridClock initialized: now 1714701602709918 us; error 30 us; skew 500 ppm
I20240503 02:00:02.716041 5807 webserver.cc:460] Webserver started at http://127.0.99.190:45381/ using document root <none> and password file <none>
I20240503 02:00:02.716578 5807 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:02.716639 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:02.716846 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:02.718474 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4a126479b44046c892f7b666e97521d4"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.718938 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4a126479b44046c892f7b666e97521d4"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.722314 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:02.725170 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:02.726460 5807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:02.726593 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4a126479b44046c892f7b666e97521d4"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.726698 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:02.755533 5807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:02.756414 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:02.756603 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:02.765893 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:39653
I20240503 02:00:02.765903 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:39653 every 8 connection(s)
I20240503 02:00:02.767062 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:02.770475 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.775485 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4: Bootstrap starting.
I20240503 02:00:02.777266 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 5807
I20240503 02:00:02.777356 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:02.777642 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.778405 5874 log.cc:824] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:02.780279 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4: No bootstrap required, opened a new log
I20240503 02:00:02.783205 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } }
I20240503 02:00:02.783390 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.783442 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a126479b44046c892f7b666e97521d4, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.783869 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } }
I20240503 02:00:02.783983 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:02.784042 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:02.784149 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.784886 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } }
I20240503 02:00:02.785199 5874 leader_election.cc:302] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a126479b44046c892f7b666e97521d4; no voters:
I20240503 02:00:02.785449 5874 leader_election.cc:288] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:02.785827 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.786028 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4a126479b44046c892f7b666e97521d4, State: Running, Role: LEADER
I20240503 02:00:02.786294 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } }
I20240503 02:00:02.786564 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:02.787361 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a126479b44046c892f7b666e97521d4. Latest consensus state: current_term: 1 leader_uuid: "4a126479b44046c892f7b666e97521d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } } }
I20240503 02:00:02.787495 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:02.788133 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a126479b44046c892f7b666e97521d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a126479b44046c892f7b666e97521d4" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39653 } } }
I20240503 02:00:02.788266 5888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:02.788288 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:02.790750 5888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:02.795593 5888 catalog_manager.cc:1347] Generated new cluster ID: 62365cc43520435db163b8bd916fce98
I20240503 02:00:02.795655 5888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:02.802769 5888 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:02.803880 5888 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:02.812321 5888 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a126479b44046c892f7b666e97521d4: Generated new TSK 0
I20240503 02:00:02.813048 5888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:02.817541 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:02.921340 5898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:02.921571 5898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:02.921661 5898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:02.924850 5898 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:02.924943 5898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:02.925035 5898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:02.928543 5898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--never_fsync=true
--heap_profile_path=/tmp/kudu.5898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:02.929351 5898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:02.930408 5898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:02.938849 5903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:02.939530 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:02.940330 5904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:02.941179 5898 server_base.cc:1034] running on GCE node
I20240503 02:00:02.941520 5898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:02.942065 5898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:02.943233 5898 hybrid_clock.cc:648] HybridClock initialized: now 1714701602943214 us; error 36 us; skew 500 ppm
I20240503 02:00:02.945108 5898 webserver.cc:460] Webserver started at http://127.0.99.129:39793/ using document root <none> and password file <none>
I20240503 02:00:02.945618 5898 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:02.945676 5898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:02.945852 5898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:02.947427 5898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "34d0a1369bba4f1583f4b00831940f83"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.947926 5898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "34d0a1369bba4f1583f4b00831940f83"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.951135 5898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:02.953167 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:02.954140 5898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:02.954239 5898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "34d0a1369bba4f1583f4b00831940f83"
format_stamp: "Formatted at 2024-05-03 02:00:02 on dist-test-slave-fvr5"
I20240503 02:00:02.954339 5898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:02.965412 5898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:02.965920 5898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:02.966080 5898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:02.966544 5898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:02.967362 5898 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:02.967408 5898 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:02.967463 5898 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:02.967494 5898 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:02.975845 5898 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46071
I20240503 02:00:02.975872 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46071 every 8 connection(s)
I20240503 02:00:02.976676 5898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:02.982708 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 5898
I20240503 02:00:02.982827 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:02.984838 6026 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39653
I20240503 02:00:02.984876 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:02.985044 6026 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:02.985431 6026 heartbeater.cc:511] Master 127.0.99.190:39653 requested a full tablet report, sending...
I20240503 02:00:02.986922 5838 ts_manager.cc:194] Registered new tserver with Master: 34d0a1369bba4f1583f4b00831940f83 (127.0.99.129:46071)
I20240503 02:00:02.988348 5838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:46847
W20240503 02:00:03.087776 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:03.088024 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:03.088094 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:03.091071 6029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:03.091150 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:03.091250 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:03.094755 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:03.095633 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:03.096750 6029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:03.105827 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.106690 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.107502 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:03.108435 6029 server_base.cc:1034] running on GCE node
I20240503 02:00:03.108801 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:03.109351 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:03.110527 6029 hybrid_clock.cc:648] HybridClock initialized: now 1714701603110506 us; error 38 us; skew 500 ppm
I20240503 02:00:03.112496 6029 webserver.cc:460] Webserver started at http://127.0.99.130:39521/ using document root <none> and password file <none>
I20240503 02:00:03.113055 6029 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:03.113114 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:03.113283 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:03.114868 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6d8b2cfb3edc46a692742b90a0bfac5f"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.115326 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6d8b2cfb3edc46a692742b90a0bfac5f"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.118441 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:03.120455 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.121376 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:03.121484 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6d8b2cfb3edc46a692742b90a0bfac5f"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.121558 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:03.138533 6029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:03.139089 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:03.139236 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:03.139724 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:03.140614 6029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:03.140671 6029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.140731 6029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:03.140758 6029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.148906 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43143
I20240503 02:00:03.148941 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43143 every 8 connection(s)
I20240503 02:00:03.149775 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:03.150110 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6029
I20240503 02:00:03.150228 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:03.152503 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:03.158785 6157 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39653
I20240503 02:00:03.159039 6157 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:03.159461 6157 heartbeater.cc:511] Master 127.0.99.190:39653 requested a full tablet report, sending...
I20240503 02:00:03.160409 5838 ts_manager.cc:194] Registered new tserver with Master: 6d8b2cfb3edc46a692742b90a0bfac5f (127.0.99.130:43143)
I20240503 02:00:03.160945 5838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52579
W20240503 02:00:03.256839 6160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:03.257066 6160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:03.257134 6160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:03.260309 6160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:03.260380 6160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:03.260469 6160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:03.263895 6160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:03.264714 6160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:03.265769 6160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:03.274421 6168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.275279 6165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.276051 6166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:03.276882 6160 server_base.cc:1034] running on GCE node
I20240503 02:00:03.277263 6160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:03.277793 6160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:03.278952 6160 hybrid_clock.cc:648] HybridClock initialized: now 1714701603278941 us; error 26 us; skew 500 ppm
I20240503 02:00:03.280799 6160 webserver.cc:460] Webserver started at http://127.0.99.131:45225/ using document root <none> and password file <none>
I20240503 02:00:03.281306 6160 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:03.281364 6160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:03.281543 6160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:03.283149 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "16b64dc1c6104767b81ffc0b01ce3939"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.283589 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "16b64dc1c6104767b81ffc0b01ce3939"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.286860 6160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:03.289024 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.289960 6160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:03.290050 6160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "16b64dc1c6104767b81ffc0b01ce3939"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.290132 6160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:03.300434 6160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:03.300928 6160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:03.301062 6160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:03.301501 6160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:03.302372 6160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:03.302417 6160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.302456 6160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:03.302487 6160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.310164 6160 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38723
I20240503 02:00:03.310186 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38723 every 8 connection(s)
I20240503 02:00:03.310947 6160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:03.317535 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6160
I20240503 02:00:03.317653 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:03.319103 6288 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39653
I20240503 02:00:03.319314 6288 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:03.319712 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:03.319845 6288 heartbeater.cc:511] Master 127.0.99.190:39653 requested a full tablet report, sending...
I20240503 02:00:03.320803 5838 ts_manager.cc:194] Registered new tserver with Master: 16b64dc1c6104767b81ffc0b01ce3939 (127.0.99.131:38723)
I20240503 02:00:03.321372 5838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:52425
W20240503 02:00:03.426105 6291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:03.426333 6291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:03.426402 6291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:03.429426 6291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:03.429498 6291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:03.429590 6291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:03.433226 6291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:03.434128 6291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:03.435250 6291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:03.443879 6296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.444696 6297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.445482 6299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:03.446259 6291 server_base.cc:1034] running on GCE node
I20240503 02:00:03.446611 6291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:03.447124 6291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:03.448282 6291 hybrid_clock.cc:648] HybridClock initialized: now 1714701603448273 us; error 24 us; skew 500 ppm
I20240503 02:00:03.450102 6291 webserver.cc:460] Webserver started at http://127.0.99.132:43147/ using document root <none> and password file <none>
I20240503 02:00:03.450599 6291 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:03.450649 6291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:03.450803 6291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:03.452422 6291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6058e1ce0f704ee08994c413ffa88564"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.452853 6291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6058e1ce0f704ee08994c413ffa88564"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.456000 6291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:03.460117 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.461066 6291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:03.461158 6291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6058e1ce0f704ee08994c413ffa88564"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.461247 6291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:03.476811 6291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:03.477342 6291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:03.477485 6291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:03.477972 6291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:03.478757 6291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:03.478801 6291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.478838 6291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:03.478852 6291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.487252 6291 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:44675
I20240503 02:00:03.487260 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:44675 every 8 connection(s)
I20240503 02:00:03.488270 6291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:03.495067 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6291
I20240503 02:00:03.495196 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:03.497195 6419 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39653
I20240503 02:00:03.497421 6419 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:03.497412 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--builtin_ntp_servers=127.0.99.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:03.497889 6419 heartbeater.cc:511] Master 127.0.99.190:39653 requested a full tablet report, sending...
I20240503 02:00:03.498880 5837 ts_manager.cc:194] Registered new tserver with Master: 6058e1ce0f704ee08994c413ffa88564 (127.0.99.132:44675)
I20240503 02:00:03.499437 5837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52091
W20240503 02:00:03.603015 6422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:03.603237 6422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:03.603309 6422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:03.606289 6422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:03.606359 6422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:03.606454 6422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:03.609916 6422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39653
--never_fsync=true
--heap_profile_path=/tmp/kudu.6422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:03.610781 6422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:03.611913 6422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:03.620280 6428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.621048 6427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:03.621786 6430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:03.622548 6422 server_base.cc:1034] running on GCE node
I20240503 02:00:03.622931 6422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:03.623428 6422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:03.624574 6422 hybrid_clock.cc:648] HybridClock initialized: now 1714701603624565 us; error 24 us; skew 500 ppm
I20240503 02:00:03.626430 6422 webserver.cc:460] Webserver started at http://127.0.99.133:40453/ using document root <none> and password file <none>
I20240503 02:00:03.626957 6422 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:03.627019 6422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:03.627183 6422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:03.628835 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3a320de142974d0ea9ea9451b5fc88f7"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.629287 6422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3a320de142974d0ea9ea9451b5fc88f7"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.632594 6422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:03.634573 6436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.635548 6422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:03.635648 6422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3a320de142974d0ea9ea9451b5fc88f7"
format_stamp: "Formatted at 2024-05-03 02:00:03 on dist-test-slave-fvr5"
I20240503 02:00:03.635735 6422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:03.653932 6422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:03.654457 6422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:03.654600 6422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:03.655071 6422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:03.655956 6422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:03.656008 6422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.656062 6422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:03.656088 6422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:03.664146 6422 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40365
I20240503 02:00:03.664217 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40365 every 8 connection(s)
I20240503 02:00:03.665030 6422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:03.672773 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6422
I20240503 02:00:03.672902 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:00:03.673151 6550 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39653
I20240503 02:00:03.673350 6550 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:03.673864 6550 heartbeater.cc:511] Master 127.0.99.190:39653 requested a full tablet report, sending...
I20240503 02:00:03.674744 5838 ts_manager.cc:194] Registered new tserver with Master: 3a320de142974d0ea9ea9451b5fc88f7 (127.0.99.133:40365)
I20240503 02:00:03.675278 5838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39791
I20240503 02:00:03.686708 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:00:03.695555 5838 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56136:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:00:03.712678 6091 tablet_service.cc:1467] Processing CreateTablet for tablet 03f0624524064cf5abdb7351b7944831 (DEFAULT_TABLE table=TestTable [id=c4b98d33241a4a368044ea9cc3d258b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.712594 6484 tablet_service.cc:1467] Processing CreateTablet for tablet 03f0624524064cf5abdb7351b7944831 (DEFAULT_TABLE table=TestTable [id=c4b98d33241a4a368044ea9cc3d258b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.712594 6222 tablet_service.cc:1467] Processing CreateTablet for tablet 03f0624524064cf5abdb7351b7944831 (DEFAULT_TABLE table=TestTable [id=c4b98d33241a4a368044ea9cc3d258b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.713500 6091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f0624524064cf5abdb7351b7944831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.713506 6484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f0624524064cf5abdb7351b7944831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.713506 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f0624524064cf5abdb7351b7944831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.718374 6567 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Bootstrap starting.
I20240503 02:00:03.719347 6568 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Bootstrap starting.
I20240503 02:00:03.720078 6569 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Bootstrap starting.
I20240503 02:00:03.720582 6567 tablet_bootstrap.cc:654] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.721084 6568 tablet_bootstrap.cc:654] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.721383 6567 log.cc:824] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:03.721689 6568 log.cc:824] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:03.722270 6569 tablet_bootstrap.cc:654] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.722963 6567 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: No bootstrap required, opened a new log
I20240503 02:00:03.722975 6569 log.cc:824] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:03.723121 6567 ts_tablet_manager.cc:1367] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:03.723246 6568 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: No bootstrap required, opened a new log
I20240503 02:00:03.723402 6568 ts_tablet_manager.cc:1367] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:03.724606 6569 tablet_bootstrap.cc:492] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: No bootstrap required, opened a new log
I20240503 02:00:03.724762 6569 ts_tablet_manager.cc:1367] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:03.726243 6567 raft_consensus.cc:348] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.726469 6567 raft_consensus.cc:374] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.726524 6567 raft_consensus.cc:729] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8b2cfb3edc46a692742b90a0bfac5f, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.726938 6567 consensus_queue.cc:260] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.726980 6568 raft_consensus.cc:348] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.727198 6568 raft_consensus.cc:374] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.727273 6568 raft_consensus.cc:729] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a320de142974d0ea9ea9451b5fc88f7, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.727768 6567 ts_tablet_manager.cc:1398] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:03.727723 6568 consensus_queue.cc:260] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.728137 6157 heartbeater.cc:503] Master 127.0.99.190:39653 was elected leader, sending a full tablet report...
I20240503 02:00:03.728544 6568 ts_tablet_manager.cc:1398] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:00:03.728932 6550 heartbeater.cc:503] Master 127.0.99.190:39653 was elected leader, sending a full tablet report...
I20240503 02:00:03.728955 6569 raft_consensus.cc:348] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.729128 6569 raft_consensus.cc:374] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.729161 6569 raft_consensus.cc:729] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b64dc1c6104767b81ffc0b01ce3939, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.729492 6569 consensus_queue.cc:260] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:03.730289 6569 ts_tablet_manager.cc:1398] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:03.730628 6288 heartbeater.cc:503] Master 127.0.99.190:39653 was elected leader, sending a full tablet report...
W20240503 02:00:03.811926 6289 tablet.cc:2369] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:03.901243 6158 tablet.cc:2369] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:03.915877 6551 tablet.cc:2369] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:03.990437 6026 heartbeater.cc:503] Master 127.0.99.190:39653 was elected leader, sending a full tablet report...
I20240503 02:00:04.047819 6574 raft_consensus.cc:482] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.048003 6574 raft_consensus.cc:504] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:04.049031 6574 leader_election.cc:288] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16b64dc1c6104767b81ffc0b01ce3939 (127.0.99.131:38723), 6d8b2cfb3edc46a692742b90a0bfac5f (127.0.99.130:43143)
I20240503 02:00:04.052501 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f0624524064cf5abdb7351b7944831" candidate_uuid: "3a320de142974d0ea9ea9451b5fc88f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b64dc1c6104767b81ffc0b01ce3939" is_pre_election: true
I20240503 02:00:04.052743 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f0624524064cf5abdb7351b7944831" candidate_uuid: "3a320de142974d0ea9ea9451b5fc88f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" is_pre_election: true
I20240503 02:00:04.052798 6242 raft_consensus.cc:2454] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a320de142974d0ea9ea9451b5fc88f7 in term 0.
I20240503 02:00:04.052987 6111 raft_consensus.cc:2454] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a320de142974d0ea9ea9451b5fc88f7 in term 0.
I20240503 02:00:04.053246 6438 leader_election.cc:302] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16b64dc1c6104767b81ffc0b01ce3939, 3a320de142974d0ea9ea9451b5fc88f7; no voters:
I20240503 02:00:04.053471 6574 raft_consensus.cc:2781] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.053539 6574 raft_consensus.cc:482] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.053567 6574 raft_consensus.cc:3037] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.054409 6574 raft_consensus.cc:504] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:04.054718 6574 leader_election.cc:288] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [CANDIDATE]: Term 1 election: Requested vote from peers 16b64dc1c6104767b81ffc0b01ce3939 (127.0.99.131:38723), 6d8b2cfb3edc46a692742b90a0bfac5f (127.0.99.130:43143)
I20240503 02:00:04.055034 6111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f0624524064cf5abdb7351b7944831" candidate_uuid: "3a320de142974d0ea9ea9451b5fc88f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f"
I20240503 02:00:04.055051 6242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f0624524064cf5abdb7351b7944831" candidate_uuid: "3a320de142974d0ea9ea9451b5fc88f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b64dc1c6104767b81ffc0b01ce3939"
I20240503 02:00:04.055138 6111 raft_consensus.cc:3037] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.055145 6242 raft_consensus.cc:3037] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.055991 6242 raft_consensus.cc:2454] T 03f0624524064cf5abdb7351b7944831 P 16b64dc1c6104767b81ffc0b01ce3939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a320de142974d0ea9ea9451b5fc88f7 in term 1.
I20240503 02:00:04.056150 6111 raft_consensus.cc:2454] T 03f0624524064cf5abdb7351b7944831 P 6d8b2cfb3edc46a692742b90a0bfac5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a320de142974d0ea9ea9451b5fc88f7 in term 1.
I20240503 02:00:04.056283 6438 leader_election.cc:302] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16b64dc1c6104767b81ffc0b01ce3939, 3a320de142974d0ea9ea9451b5fc88f7; no voters:
I20240503 02:00:04.056468 6574 raft_consensus.cc:2781] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.056685 6574 raft_consensus.cc:686] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a320de142974d0ea9ea9451b5fc88f7, State: Running, Role: LEADER
I20240503 02:00:04.056993 6574 consensus_queue.cc:237] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } }
I20240503 02:00:04.059830 5838 catalog_manager.cc:5484] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a320de142974d0ea9ea9451b5fc88f7 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "3a320de142974d0ea9ea9451b5fc88f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 } } peers { permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 } } peers { permanent_uuid: "3a320de142974d0ea9ea9451b5fc88f7" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40365 } } }
I20240503 02:00:04.066488 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:00:04.068008 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 34d0a1369bba4f1583f4b00831940f83 to finish bootstrapping
I20240503 02:00:04.073426 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6d8b2cfb3edc46a692742b90a0bfac5f to finish bootstrapping
I20240503 02:00:04.078071 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 16b64dc1c6104767b81ffc0b01ce3939 to finish bootstrapping
I20240503 02:00:04.082389 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6058e1ce0f704ee08994c413ffa88564 to finish bootstrapping
I20240503 02:00:04.087234 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3a320de142974d0ea9ea9451b5fc88f7 to finish bootstrapping
I20240503 02:00:04.271170 6222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.271654 6484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:04.273800 5960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.274395 6353 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.281231 6091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.363953 5960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.364486 6222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.365036 6484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:04.368062 6091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.373612 6353 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.501909 6419 heartbeater.cc:503] Master 127.0.99.190:39653 was elected leader, sending a full tablet report...
I20240503 02:00:04.514187 6574 consensus_queue.cc:1035] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8b2cfb3edc46a692742b90a0bfac5f" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.519330 6574 consensus_queue.cc:1035] T 03f0624524064cf5abdb7351b7944831 P 3a320de142974d0ea9ea9451b5fc88f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b64dc1c6104767b81ffc0b01ce3939" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.577996 6222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.579622 5960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.580001 6484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:04.580785 6353 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.587091 6091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.670004 6222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.670801 6484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:04.670897 6353 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.671784 6091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.683988 5960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.782048 5960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.783247 6222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.783690 6353 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.788422 6091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:04.793323 6484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:04.822057 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 5898
I20240503 02:00:04.827780 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6029
I20240503 02:00:04.833559 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6160
I20240503 02:00:04.839238 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6291
I20240503 02:00:04.844437 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6422
I20240503 02:00:04.849975 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 5807
2024-05-03T02:00:04Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2299 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2299 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240503 02:00:04.872402 398 test_util.cc:268] Using random seed: 336453837
I20240503 02:00:04.873153 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:04.873194 398 ts_itest-base.cc:113] --------------
I20240503 02:00:04.873215 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:00:04.873231 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 02:00:04.873258 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:04Z Disabled control of system clock
I20240503 02:00:04.880832 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36947
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36947
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:00:04.983608 6760 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:00:04.983855 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:04.983881 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:04.983916 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:04.986801 6760 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:04.986858 6760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:04.986881 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:04.986900 6760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:04.986919 6760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:04.990389 6760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36947
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:04.991165 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:04.992213 6760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:04.999686 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.000483 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.001230 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.002064 6760 server_base.cc:1034] running on GCE node
I20240503 02:00:05.002511 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.003376 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.004536 6760 hybrid_clock.cc:648] HybridClock initialized: now 1714701605004522 us; error 29 us; skew 500 ppm
I20240503 02:00:05.006356 6760 webserver.cc:460] Webserver started at http://127.0.99.190:35681/ using document root <none> and password file <none>
I20240503 02:00:05.006873 6760 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.006927 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.007115 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.008721 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cc32e3225a044351a09cc29d3e9fff23"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.009176 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cc32e3225a044351a09cc29d3e9fff23"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.012287 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:05.014321 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.015235 6760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.015357 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cc32e3225a044351a09cc29d3e9fff23"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.015435 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.042886 6760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.043514 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.043668 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.050073 6760 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36947
I20240503 02:00:05.050087 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36947 every 8 connection(s)
I20240503 02:00:05.050997 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:05.053460 6827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.055732 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6760
I20240503 02:00:05.055866 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:05.058756 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: Bootstrap starting.
I20240503 02:00:05.060732 6827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.061370 6827 log.cc:824] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:05.063205 6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: No bootstrap required, opened a new log
I20240503 02:00:05.065543 6827 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } }
I20240503 02:00:05.065713 6827 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.065747 6827 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc32e3225a044351a09cc29d3e9fff23, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.066107 6827 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } }
I20240503 02:00:05.066221 6827 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:05.066264 6827 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:05.066344 6827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.066984 6827 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } }
I20240503 02:00:05.067253 6827 leader_election.cc:302] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc32e3225a044351a09cc29d3e9fff23; no voters:
I20240503 02:00:05.067469 6827 leader_election.cc:288] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:05.067571 6832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.067777 6832 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [term 1 LEADER]: Becoming Leader. State: Replica: cc32e3225a044351a09cc29d3e9fff23, State: Running, Role: LEADER
I20240503 02:00:05.068053 6832 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } }
I20240503 02:00:05.068449 6827 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:05.069962 6833 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc32e3225a044351a09cc29d3e9fff23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } } }
I20240503 02:00:05.070084 6833 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:05.069965 6834 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc32e3225a044351a09cc29d3e9fff23. Latest consensus state: current_term: 1 leader_uuid: "cc32e3225a044351a09cc29d3e9fff23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc32e3225a044351a09cc29d3e9fff23" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36947 } } }
I20240503 02:00:05.070824 6834 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23 [sys.catalog]: This master's current role is: LEADER
W20240503 02:00:05.073073 6847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:00:05.073137 6847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:00:05.073211 6846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:05.073943 6846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:05.078696 6846 catalog_manager.cc:1347] Generated new cluster ID: 553daad504614114b25b08467be7b505
I20240503 02:00:05.078761 6846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:05.091456 6846 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:05.092600 6846 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:05.100057 6846 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cc32e3225a044351a09cc29d3e9fff23: Generated new TSK 0
I20240503 02:00:05.100804 6846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:05.109709 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:05.213835 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.214087 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.214157 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.217257 6851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:05.217332 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.217422 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:05.221171 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.222069 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.223155 6851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.231582 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.232498 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.233230 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.234061 6851 server_base.cc:1034] running on GCE node
I20240503 02:00:05.234431 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.234968 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.236121 6851 hybrid_clock.cc:648] HybridClock initialized: now 1714701605236111 us; error 26 us; skew 500 ppm
I20240503 02:00:05.237882 6851 webserver.cc:460] Webserver started at http://127.0.99.129:46459/ using document root <none> and password file <none>
I20240503 02:00:05.238418 6851 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.238478 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.238646 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.240303 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.240753 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.243860 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:05.246037 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.247073 6851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.247174 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.247257 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.258500 6851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.259074 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.259228 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.259706 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.260622 6851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.260668 6851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.260727 6851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.260756 6851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.268870 6851 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:45253
I20240503 02:00:05.268951 6978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:45253 every 8 connection(s)
I20240503 02:00:05.269801 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:05.274854 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6851
I20240503 02:00:05.274966 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:05.277101 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:05.278458 6979 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:05.278664 6979 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.279062 6979 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:05.280440 6787 ts_manager.cc:194] Registered new tserver with Master: faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:05.282121 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:53457
W20240503 02:00:05.380692 6982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.380887 6982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.380928 6982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.383818 6982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:05.383873 6982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.383936 6982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:05.387356 6982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.6982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.388141 6982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.389166 6982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.397393 6990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.398197 6987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.398931 6988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.399726 6982 server_base.cc:1034] running on GCE node
I20240503 02:00:05.400096 6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.400619 6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.401799 6982 hybrid_clock.cc:648] HybridClock initialized: now 1714701605401777 us; error 36 us; skew 500 ppm
I20240503 02:00:05.403848 6982 webserver.cc:460] Webserver started at http://127.0.99.130:36603/ using document root <none> and password file <none>
I20240503 02:00:05.404376 6982 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.404436 6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.404603 6982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.406194 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.406646 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.409826 6982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:05.411885 6996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.412873 6982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.412981 6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.413064 6982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.429646 6982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.430267 6982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.430431 6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.431015 6982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.431865 6982 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.431908 6982 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.431954 6982 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.431984 6982 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.440482 6982 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:46587
I20240503 02:00:05.440518 7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:46587 every 8 connection(s)
I20240503 02:00:05.441339 6982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:05.443158 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 6982
I20240503 02:00:05.443264 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:05.445286 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:05.449795 7110 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:05.450024 7110 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.450441 7110 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:05.451423 6787 ts_manager.cc:194] Registered new tserver with Master: 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:05.452278 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56053
W20240503 02:00:05.549880 7113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.550086 7113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.550127 7113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.553133 7113 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:05.553191 7113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.553256 7113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:05.556679 7113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.7113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.557451 7113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.558513 7113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.567191 7121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.568020 7119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.568815 7118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.569660 7113 server_base.cc:1034] running on GCE node
I20240503 02:00:05.570055 7113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.570668 7113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.571818 7113 hybrid_clock.cc:648] HybridClock initialized: now 1714701605571769 us; error 57 us; skew 500 ppm
I20240503 02:00:05.573766 7113 webserver.cc:460] Webserver started at http://127.0.99.131:46463/ using document root <none> and password file <none>
I20240503 02:00:05.574349 7113 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.574410 7113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.574591 7113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.576246 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a648a1aaf5eb44faa930c687aac928de"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.576700 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a648a1aaf5eb44faa930c687aac928de"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.579916 7113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:05.582002 7127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.582940 7113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.583043 7113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a648a1aaf5eb44faa930c687aac928de"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.583122 7113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.602167 7113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.602757 7113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.602911 7113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.603389 7113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.604271 7113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.604317 7113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.604365 7113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.604399 7113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.612857 7113 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45815
I20240503 02:00:05.612973 7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45815 every 8 connection(s)
I20240503 02:00:05.613721 7113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:05.621528 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 7113
I20240503 02:00:05.621618 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:05.621822 7241 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:05.622041 7241 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.622613 7241 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:05.623469 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:05.623507 6787 ts_manager.cc:194] Registered new tserver with Master: a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:05.624205 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:40859
W20240503 02:00:05.729612 7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.729846 7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.729888 7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.732901 7244 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:05.732957 7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.733052 7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:05.736563 7244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.7244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.737387 7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.738476 7244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.746997 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.747838 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.748589 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.749384 7244 server_base.cc:1034] running on GCE node
I20240503 02:00:05.749737 7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.750278 7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.751421 7244 hybrid_clock.cc:648] HybridClock initialized: now 1714701605751406 us; error 38 us; skew 500 ppm
I20240503 02:00:05.753188 7244 webserver.cc:460] Webserver started at http://127.0.99.132:44797/ using document root <none> and password file <none>
I20240503 02:00:05.753710 7244 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.753764 7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.753932 7244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.755535 7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "07bf298e40944e249d6ed0d50087aff7"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.756026 7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "07bf298e40944e249d6ed0d50087aff7"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.759135 7244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 02:00:05.761251 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.762250 7244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.762353 7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "07bf298e40944e249d6ed0d50087aff7"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.762430 7244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.773569 7244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.774143 7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.774284 7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.774770 7244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.775594 7244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.775642 7244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.775697 7244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.775730 7244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.784248 7244 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41379
I20240503 02:00:05.784274 7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41379 every 8 connection(s)
I20240503 02:00:05.785104 7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:05.788905 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 7244
I20240503 02:00:05.789005 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:05.790932 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:05.793648 7372 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:05.793849 7372 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.794255 7372 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:05.795167 6787 ts_manager.cc:194] Registered new tserver with Master: 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379)
I20240503 02:00:05.795984 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:52553
W20240503 02:00:05.894064 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:05.894272 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:05.894313 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:05.897303 7375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:05.897364 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:05.897429 7375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:05.900898 7375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.7375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:05.901674 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:05.902710 7375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:05.911365 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.912196 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:05.912986 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:05.913794 7375 server_base.cc:1034] running on GCE node
I20240503 02:00:05.914141 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:05.914639 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:05.915799 7375 hybrid_clock.cc:648] HybridClock initialized: now 1714701605915777 us; error 45 us; skew 500 ppm
I20240503 02:00:05.917646 7375 webserver.cc:460] Webserver started at http://127.0.99.133:40743/ using document root <none> and password file <none>
I20240503 02:00:05.918177 7375 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:05.918237 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:05.918406 7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:05.920099 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.920542 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.923609 7375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:05.925714 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.926668 7375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:05.926767 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:05.926846 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:05.950820 7375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:05.951507 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:05.951678 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:05.952236 7375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:05.953190 7375 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:05.953243 7375 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.953305 7375 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:05.953331 7375 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:05.961423 7375 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35021
I20240503 02:00:05.961467 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35021 every 8 connection(s)
I20240503 02:00:05.962330 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:05.966383 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 7375
I20240503 02:00:05.966519 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:00:05.968494 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:05.970999 7503 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:05.971187 7503 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:05.971575 7503 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:05.972546 6787 ts_manager.cc:194] Registered new tserver with Master: 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
I20240503 02:00:05.973373 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:32833
W20240503 02:00:06.075284 7506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.075480 7506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.075522 7506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.078467 7506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:06.078527 7506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.078590 7506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:06.082263 7506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.7506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.083092 7506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.084235 7506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.092677 7512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.093525 7511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.094338 7514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.095192 7506 server_base.cc:1034] running on GCE node
I20240503 02:00:06.095535 7506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.096105 7506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.097254 7506 hybrid_clock.cc:648] HybridClock initialized: now 1714701606097239 us; error 25 us; skew 500 ppm
I20240503 02:00:06.099050 7506 webserver.cc:460] Webserver started at http://127.0.99.134:38943/ using document root <none> and password file <none>
I20240503 02:00:06.099572 7506 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.099633 7506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.099845 7506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.101446 7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "591f812606bf438fbb9bd0a977c678e6"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.101897 7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "591f812606bf438fbb9bd0a977c678e6"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.105041 7506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:06.107033 7520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.108044 7506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:06.108145 7506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "591f812606bf438fbb9bd0a977c678e6"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.108222 7506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.124228 7506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.124846 7506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.125010 7506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.125514 7506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:06.126386 7506 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:06.126435 7506 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.126497 7506 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:06.126523 7506 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.134955 7506 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:45311
I20240503 02:00:06.135020 7633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:45311 every 8 connection(s)
I20240503 02:00:06.136117 7506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:06.144030 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 7506
I20240503 02:00:06.144155 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:00:06.144613 7634 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:06.144834 7634 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:06.145243 7634 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:06.146081 6787 ts_manager.cc:194] Registered new tserver with Master: 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311)
I20240503 02:00:06.146185 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:06.146646 6787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:54579
W20240503 02:00:06.251569 7637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:06.251782 7637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:06.251848 7637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:06.254851 7637 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:06.254921 7637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:06.254997 7637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:06.258467 7637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.7637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:06.259285 7637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:06.260432 7637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:06.268821 7643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.269595 7642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:06.270324 7645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:06.271173 7637 server_base.cc:1034] running on GCE node
I20240503 02:00:06.271493 7637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:06.272039 7637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:06.273183 7637 hybrid_clock.cc:648] HybridClock initialized: now 1714701606273151 us; error 49 us; skew 500 ppm
I20240503 02:00:06.275010 7637 webserver.cc:460] Webserver started at http://127.0.99.135:42375/ using document root <none> and password file <none>
I20240503 02:00:06.275519 7637 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:06.275579 7637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:06.275779 7637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:06.277282 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "12549acb31c04923a95111409fc4bad7"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.277716 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "12549acb31c04923a95111409fc4bad7"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.280758 7637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:06.282680 7651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.283560 7637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:06.283658 7637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "12549acb31c04923a95111409fc4bad7"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:06.283746 7637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:06.284224 6979 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:06.311581 7637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:06.312237 7637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:06.312393 7637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:06.312919 7637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:06.313755 7637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:06.313798 7637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.313836 7637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:06.313858 7637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:06.322510 7637 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41723
I20240503 02:00:06.322536 7764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41723 every 8 connection(s)
I20240503 02:00:06.323415 7637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:06.331259 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 7637
I20240503 02:00:06.331374 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:00:06.331710 7765 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:06.331959 7765 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:06.332396 7765 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:06.333307 6790 ts_manager.cc:194] Registered new tserver with Master: 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:06.333933 6790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:52183
I20240503 02:00:06.345410 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:00:06.353808 6790 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:00:06.371608 7437 tablet_service.cc:1467] Processing CreateTablet for tablet b47074508be1424f9c1686bcc6526698 (DEFAULT_TABLE table=TestTable [id=a0eda56e70d74d2c8d48d163c118b2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.371608 7699 tablet_service.cc:1467] Processing CreateTablet for tablet b47074508be1424f9c1686bcc6526698 (DEFAULT_TABLE table=TestTable [id=a0eda56e70d74d2c8d48d163c118b2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:06.372467 7699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47074508be1424f9c1686bcc6526698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.372465 7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47074508be1424f9c1686bcc6526698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:06.376909 7781 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:06.377683 7782 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap starting.
I20240503 02:00:06.378710 7781 tablet_bootstrap.cc:654] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.379354 7782 tablet_bootstrap.cc:654] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:06.379438 7781 log.cc:824] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:06.380026 7782 log.cc:824] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:06.380985 7781 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: No bootstrap required, opened a new log
I20240503 02:00:06.381165 7781 ts_tablet_manager.cc:1367] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:06.381318 7782 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: No bootstrap required, opened a new log
I20240503 02:00:06.381466 7782 ts_tablet_manager.cc:1367] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:06.384290 7782 raft_consensus.cc:348] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.384284 7781 raft_consensus.cc:348] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.384442 7782 raft_consensus.cc:374] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.384447 7781 raft_consensus.cc:374] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:06.384485 7782 raft_consensus.cc:729] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.384503 7781 raft_consensus.cc:729] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:06.384868 7782 consensus_queue.cc:260] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.384871 7781 consensus_queue.cc:260] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.385460 7765 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:06.385468 7503 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:06.385648 7782 ts_tablet_manager.cc:1398] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:06.385670 7781 ts_tablet_manager.cc:1398] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:00:06.454059 7110 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
W20240503 02:00:06.463337 7504 tablet.cc:2369] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:06.570673 7785 raft_consensus.cc:482] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:06.570878 7785 raft_consensus.cc:504] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.571902 7785 leader_election.cc:288] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
W20240503 02:00:06.574218 7766 tablet.cc:2369] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:06.576735 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47074508be1424f9c1686bcc6526698" candidate_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12549acb31c04923a95111409fc4bad7" is_pre_election: true
I20240503 02:00:06.577064 7719 raft_consensus.cc:2454] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d67357ff41b39ea5317b0874cd4c in term 0.
I20240503 02:00:06.577600 7392 leader_election.cc:302] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12549acb31c04923a95111409fc4bad7, 8ac2d67357ff41b39ea5317b0874cd4c; no voters:
I20240503 02:00:06.577972 7785 raft_consensus.cc:2781] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:06.578063 7785 raft_consensus.cc:482] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:06.578116 7785 raft_consensus.cc:3037] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.579046 7785 raft_consensus.cc:504] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.579337 7785 leader_election.cc:288] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 1 election: Requested vote from peers 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:06.579741 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47074508be1424f9c1686bcc6526698" candidate_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12549acb31c04923a95111409fc4bad7"
I20240503 02:00:06.579923 7719 raft_consensus.cc:3037] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:06.581234 7719 raft_consensus.cc:2454] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d67357ff41b39ea5317b0874cd4c in term 1.
I20240503 02:00:06.581591 7392 leader_election.cc:302] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12549acb31c04923a95111409fc4bad7, 8ac2d67357ff41b39ea5317b0874cd4c; no voters:
I20240503 02:00:06.581799 7785 raft_consensus.cc:2781] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:06.581995 7785 raft_consensus.cc:686] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 LEADER]: Becoming Leader. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Running, Role: LEADER
I20240503 02:00:06.582306 7785 consensus_queue.cc:237] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:06.585134 6791 catalog_manager.cc:5484] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133). New cstate: current_term: 1 leader_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:06.609846 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:00:06.611163 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver faa3c52a3f9b4bbda3ca9743ed432a08 to finish bootstrapping
I20240503 02:00:06.617107 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 69a1ad6834cc4c0dbd0436227b1f9dbb to finish bootstrapping
I20240503 02:00:06.622305 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a648a1aaf5eb44faa930c687aac928de to finish bootstrapping
I20240503 02:00:06.626401 7241 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:06.627489 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 07bf298e40944e249d6ed0d50087aff7 to finish bootstrapping
I20240503 02:00:06.632524 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8ac2d67357ff41b39ea5317b0874cd4c to finish bootstrapping
I20240503 02:00:06.637085 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 591f812606bf438fbb9bd0a977c678e6 to finish bootstrapping
I20240503 02:00:06.642166 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12549acb31c04923a95111409fc4bad7 to finish bootstrapping
I20240503 02:00:06.646343 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 7244
I20240503 02:00:06.652428 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 7375
I20240503 02:00:06.657927 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 7506
I20240503 02:00:06.663029 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 7637
I20240503 02:00:10.419840 6791 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:10.432669 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9d6747dd9146f99ec8e06ef37b0c87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.432673 7043 tablet_service.cc:1467] Processing CreateTablet for tablet c77c0d649d2c4ae39e0ade14fbe06c9e (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.432953 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9d6747dd9146f99ec8e06ef37b0c87 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.432950 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 34c7315583b84d67853e2d5de4d390d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.433637 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d6747dd9146f99ec8e06ef37b0c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.433633 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c0d649d2c4ae39e0ade14fbe06c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.433689 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c7315583b84d67853e2d5de4d390d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.436064 7044 tablet_service.cc:1467] Processing CreateTablet for tablet 34c7315583b84d67853e2d5de4d390d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.436156 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d6747dd9146f99ec8e06ef37b0c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.436293 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c7315583b84d67853e2d5de4d390d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.432670 7174 tablet_service.cc:1467] Processing CreateTablet for tablet c77c0d649d2c4ae39e0ade14fbe06c9e (DEFAULT_TABLE table=rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:10.439524 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c0d649d2c4ae39e0ade14fbe06c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:10.440033 7808 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:10.440081 7806 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:10.441471 7807 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:10.442135 7806 tablet_bootstrap.cc:654] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.442135 7808 tablet_bootstrap.cc:654] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.442912 7808 log.cc:824] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:10.442912 7806 log.cc:824] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:10.442950 7807 tablet_bootstrap.cc:654] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.443634 7807 log.cc:824] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:10.444516 7808 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:10.444516 7806 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:10.444711 7806 ts_tablet_manager.cc:1367] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:10.444711 7808 ts_tablet_manager.cc:1367] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:00:10.445116 7807 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:10.445261 7807 ts_tablet_manager.cc:1367] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:10.447952 7807 raft_consensus.cc:348] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.448113 7807 raft_consensus.cc:374] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.448184 7807 raft_consensus.cc:729] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.448185 7806 raft_consensus.cc:348] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.448371 7806 raft_consensus.cc:374] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.448422 7806 raft_consensus.cc:729] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.448395 7808 raft_consensus.cc:348] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.448559 7808 raft_consensus.cc:374] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.448608 7808 raft_consensus.cc:729] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.448599 7807 consensus_queue.cc:260] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.448877 7806 consensus_queue.cc:260] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.448927 7808 consensus_queue.cc:260] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.449685 7806 ts_tablet_manager.cc:1398] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:10.449685 7808 ts_tablet_manager.cc:1398] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:10.449934 7806 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:10.450479 7808 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:10.450829 7806 tablet_bootstrap.cc:654] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.451942 7807 ts_tablet_manager.cc:1398] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:00:10.452196 7807 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:10.452504 7808 tablet_bootstrap.cc:654] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.452993 7807 tablet_bootstrap.cc:654] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:10.453634 7806 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:10.453666 7808 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:10.453718 7806 ts_tablet_manager.cc:1367] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:10.453738 7808 ts_tablet_manager.cc:1367] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:10.454150 7806 raft_consensus.cc:348] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.454200 7807 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:10.454217 7806 raft_consensus.cc:374] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.454237 7806 raft_consensus.cc:729] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.454273 7807 ts_tablet_manager.cc:1367] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:10.454248 7808 raft_consensus.cc:348] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.454330 7808 raft_consensus.cc:374] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.454329 7806 consensus_queue.cc:260] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.454367 7808 raft_consensus.cc:729] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.454483 7808 consensus_queue.cc:260] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.454555 7806 ts_tablet_manager.cc:1398] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:10.454699 7808 ts_tablet_manager.cc:1398] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:10.454747 7807 raft_consensus.cc:348] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.454859 7807 raft_consensus.cc:374] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:10.454900 7807 raft_consensus.cc:729] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:10.455025 7807 consensus_queue.cc:260] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.455297 7807 ts_tablet_manager.cc:1398] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 02:00:10.524077 6980 tablet.cc:2369] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:10.556751 7812 raft_consensus.cc:482] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.556951 7812 raft_consensus.cc:504] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.558023 7812 leader_election.cc:288] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:10.561503 7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:10.561784 7064 raft_consensus.cc:2454] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 0.
I20240503 02:00:10.562208 7131 leader_election.cc:302] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:10.562451 7812 raft_consensus.cc:2781] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.562523 7812 raft_consensus.cc:482] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.562563 7812 raft_consensus.cc:3037] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.563431 7812 raft_consensus.cc:504] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.563676 7812 leader_election.cc:288] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:10.564083 7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:10.564205 7064 raft_consensus.cc:3037] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.565107 7064 raft_consensus.cc:2454] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 1.
I20240503 02:00:10.565387 7131 leader_election.cc:302] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:10.565549 7812 raft_consensus.cc:2781] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.565776 7812 raft_consensus.cc:686] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:10.566115 7812 consensus_queue.cc:237] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:10.568696 6787 catalog_manager.cc:5484] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 0 to 1, leader changed from <none> to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
I20240503 02:00:10.584757 7813 raft_consensus.cc:482] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.584944 7813 raft_consensus.cc:504] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.585911 7813 leader_election.cc:288] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:10.589443 7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c7315583b84d67853e2d5de4d390d5" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:10.589568 7064 raft_consensus.cc:2454] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:10.589861 6869 leader_election.cc:302] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:10.590102 7813 raft_consensus.cc:2781] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.590173 7813 raft_consensus.cc:482] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.590231 7813 raft_consensus.cc:3037] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.591099 7813 raft_consensus.cc:504] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.591356 7813 leader_election.cc:288] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:10.591718 7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c7315583b84d67853e2d5de4d390d5" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:10.591885 7064 raft_consensus.cc:3037] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.592581 7064 raft_consensus.cc:2454] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:10.592829 6869 leader_election.cc:302] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:10.592985 7813 raft_consensus.cc:2781] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.593194 7813 raft_consensus.cc:686] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:10.593484 7813 consensus_queue.cc:237] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:10.595904 6787 catalog_manager.cc:5484] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } }
W20240503 02:00:10.618139 7242 tablet.cc:2369] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:10.696035 7111 tablet.cc:2369] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:10.805557 7812 raft_consensus.cc:482] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:10.805691 7812 raft_consensus.cc:504] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.806130 7812 leader_election.cc:288] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:10.809441 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d6747dd9146f99ec8e06ef37b0c87" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" is_pre_election: true
I20240503 02:00:10.809636 6933 raft_consensus.cc:2454] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 0.
I20240503 02:00:10.809974 7128 leader_election.cc:302] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:10.810142 7812 raft_consensus.cc:2781] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:10.810199 7812 raft_consensus.cc:482] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:10.810235 7812 raft_consensus.cc:3037] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.810954 7812 raft_consensus.cc:504] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.811185 7812 leader_election.cc:288] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:10.811522 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9d6747dd9146f99ec8e06ef37b0c87" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:10.811627 6933 raft_consensus.cc:3037] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:10.812407 6933 raft_consensus.cc:2454] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 1.
I20240503 02:00:10.812680 7128 leader_election.cc:302] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:10.812857 7812 raft_consensus.cc:2781] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:10.812945 7812 raft_consensus.cc:686] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:10.813213 7812 consensus_queue.cc:237] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:10.814899 6787 catalog_manager.cc:5484] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 0 to 1, leader changed from <none> to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 1 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } }
I20240503 02:00:10.944685 7812 consensus_queue.cc:1035] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:00:10.981206 7825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:10.981431 7825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:10.984424 7825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:00:11.072597 6933 raft_consensus.cc:1261] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.073274 7812 consensus_queue.cc:1035] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.075378 7064 raft_consensus.cc:1261] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.076588 7813 consensus_queue.cc:1035] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.087152 7858 mvcc.cc:205] Tried to move back new op lower bound from 7023417798962642944 to 7023417796992811008. Current Snapshot: MvccSnapshot[applied={T|T < 7023417798945554432}]
I20240503 02:00:11.092993 7828 mvcc.cc:205] Tried to move back new op lower bound from 7023417798962642944 to 7023417796992811008. Current Snapshot: MvccSnapshot[applied={T|T < 7023417798945554432}]
INSERT report
rows total: 2000
time total: 87.5159 ms
time per row: 0.0437579 ms
I20240503 02:00:11.117209 6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:11.122393 6911 tablet_service.cc:1467] Processing CreateTablet for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.122419 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 2408d49d82e34cd48ce55337221c3757 (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.122563 7043 tablet_service.cc:1467] Processing CreateTablet for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.122728 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef695f025d7c479b98b9e8b6ee4fc4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.122840 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef695f025d7c479b98b9e8b6ee4fc4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.122947 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2408d49d82e34cd48ce55337221c3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.123235 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 2408d49d82e34cd48ce55337221c3757 (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.123476 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2408d49d82e34cd48ce55337221c3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.125087 7877 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:11.125445 7043 tablet_service.cc:1467] Processing CreateTablet for tablet a39950d966ac43b890d417c3faa5992b (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.125706 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39950d966ac43b890d417c3faa5992b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.125705 7878 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:11.125838 7174 tablet_service.cc:1467] Processing CreateTablet for tablet a39950d966ac43b890d417c3faa5992b (DEFAULT_TABLE table=rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.126084 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39950d966ac43b890d417c3faa5992b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.126185 7877 tablet_bootstrap.cc:654] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.127020 7878 tablet_bootstrap.cc:654] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.127513 7876 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:11.128274 7876 tablet_bootstrap.cc:654] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.128376 7878 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:11.128445 7878 ts_tablet_manager.cc:1367] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.128892 7878 raft_consensus.cc:348] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.129000 7878 raft_consensus.cc:374] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.129042 7878 raft_consensus.cc:729] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.129168 7878 consensus_queue.cc:260] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.129240 7876 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:11.129307 7876 ts_tablet_manager.cc:1367] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.129642 7876 raft_consensus.cc:348] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.129719 7876 raft_consensus.cc:374] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.129746 7876 raft_consensus.cc:729] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.129853 7876 consensus_queue.cc:260] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.129896 7878 ts_tablet_manager.cc:1398] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.130070 7878 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:11.130111 7876 ts_tablet_manager.cc:1398] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.130292 7876 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:11.130549 7877 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:11.130640 7877 ts_tablet_manager.cc:1367] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:11.131042 7876 tablet_bootstrap.cc:654] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.131007 7877 raft_consensus.cc:348] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.131078 7877 raft_consensus.cc:374] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.131098 7877 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.131217 7877 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.131450 7877 ts_tablet_manager.cc:1398] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.131598 7877 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:11.132328 7877 tablet_bootstrap.cc:654] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.133476 7877 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:11.133554 7877 ts_tablet_manager.cc:1367] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:11.133594 7876 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:11.133644 7876 ts_tablet_manager.cc:1367] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:11.133937 7876 raft_consensus.cc:348] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.134024 7876 raft_consensus.cc:374] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.134060 7876 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.134012 7877 raft_consensus.cc:348] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.134097 7877 raft_consensus.cc:374] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.134128 7877 raft_consensus.cc:729] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.134229 7877 consensus_queue.cc:260] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.134311 7876 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.134701 7877 ts_tablet_manager.cc:1398] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.135428 7876 ts_tablet_manager.cc:1398] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:11.135525 7878 tablet_bootstrap.cc:654] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.136634 7878 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:11.136723 7878 ts_tablet_manager.cc:1367] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 02:00:11.137045 7878 raft_consensus.cc:348] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.137133 7878 raft_consensus.cc:374] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.137166 7878 raft_consensus.cc:729] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.137284 7878 consensus_queue.cc:260] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.137517 7878 ts_tablet_manager.cc:1398] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.260715 7822 raft_consensus.cc:482] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.260850 7822 raft_consensus.cc:504] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.261235 7822 leader_election.cc:288] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.264760 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:11.264976 7195 raft_consensus.cc:2454] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:11.265345 6868 leader_election.cc:302] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.265551 7822 raft_consensus.cc:2781] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.265612 7822 raft_consensus.cc:482] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.265641 7822 raft_consensus.cc:3037] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.266496 7822 raft_consensus.cc:504] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.266788 7822 leader_election.cc:288] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.267158 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:11.267266 7195 raft_consensus.cc:3037] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.268098 7195 raft_consensus.cc:2454] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:11.268369 6868 leader_election.cc:302] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.268530 7822 raft_consensus.cc:2781] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.268610 7822 raft_consensus.cc:686] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:11.268772 7822 consensus_queue.cc:237] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.273450 6787 catalog_manager.cc:5484] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
I20240503 02:00:11.347534 7814 raft_consensus.cc:482] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.347673 7814 raft_consensus.cc:504] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.348683 7814 leader_election.cc:288] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.351970 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39950d966ac43b890d417c3faa5992b" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:11.352097 7195 raft_consensus.cc:2454] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 0.
I20240503 02:00:11.352401 6999 leader_election.cc:302] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:11.352615 7814 raft_consensus.cc:2781] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.352677 7814 raft_consensus.cc:482] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.352726 7814 raft_consensus.cc:3037] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.353534 7814 raft_consensus.cc:504] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.353778 7814 leader_election.cc:288] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.354094 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39950d966ac43b890d417c3faa5992b" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:11.354218 7195 raft_consensus.cc:3037] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.354928 7195 raft_consensus.cc:2454] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 1.
I20240503 02:00:11.355161 6999 leader_election.cc:302] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:11.355301 7814 raft_consensus.cc:2781] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.355489 7814 raft_consensus.cc:686] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:11.355813 7814 consensus_queue.cc:237] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.357748 6787 catalog_manager.cc:5484] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } }
I20240503 02:00:11.413226 7822 raft_consensus.cc:482] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.413359 7822 raft_consensus.cc:504] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.413710 7822 leader_election.cc:288] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:11.414217 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:11.414362 7063 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:11.414647 6869 leader_election.cc:302] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.414825 7822 raft_consensus.cc:2781] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.414880 7822 raft_consensus.cc:482] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.414928 7822 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.415781 7822 raft_consensus.cc:504] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.416035 7822 leader_election.cc:288] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:11.416409 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:11.416517 7063 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.417412 7063 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:11.417713 6869 leader_election.cc:302] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.417960 7822 raft_consensus.cc:2781] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.418062 7822 raft_consensus.cc:686] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:11.418195 7822 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:11.419595 6787 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } }
W20240503 02:00:11.597456 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:11.597692 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:11.600708 7891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:00:11.691642 7195 raft_consensus.cc:1261] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.691933 7194 raft_consensus.cc:1261] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.692373 7814 consensus_queue.cc:1035] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.692584 7063 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.692642 7822 consensus_queue.cc:1035] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.693315 7822 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:11.704035 7926 mvcc.cc:205] Tried to move back new op lower bound from 7023417801483554816 to 7023417799757791232. Current Snapshot: MvccSnapshot[applied={T|T < 7023417801483554816}]
INSERT report
rows total: 2000
time total: 94.5732 ms
time per row: 0.0472866 ms
I20240503 02:00:11.739954 6787 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:11.745709 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad4302f3e0d4e1191aff07a2f39765a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.746017 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4302f3e0d4e1191aff07a2f39765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.745913 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 8af5c8713282498abc8cf3606a7d2279 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.746260 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af5c8713282498abc8cf3606a7d2279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.746999 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad4302f3e0d4e1191aff07a2f39765a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.747253 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4302f3e0d4e1191aff07a2f39765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.748057 7942 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:11.748881 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 77ca27caa72e415ebed755eb8a11d601 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.749051 7942 tablet_bootstrap.cc:654] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.749125 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca27caa72e415ebed755eb8a11d601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.749488 7043 tablet_service.cc:1467] Processing CreateTablet for tablet 8af5c8713282498abc8cf3606a7d2279 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.749760 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af5c8713282498abc8cf3606a7d2279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.750420 7942 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:11.750505 7942 ts_tablet_manager.cc:1367] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:11.750950 7942 raft_consensus.cc:348] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.751067 7942 raft_consensus.cc:374] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.751152 7942 raft_consensus.cc:729] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.751314 7942 consensus_queue.cc:260] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.751607 7942 ts_tablet_manager.cc:1398] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.751801 7942 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:11.751964 7945 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:11.752126 7044 tablet_service.cc:1467] Processing CreateTablet for tablet 77ca27caa72e415ebed755eb8a11d601 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:11.752347 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca27caa72e415ebed755eb8a11d601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:11.752640 7942 tablet_bootstrap.cc:654] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.753152 7945 tablet_bootstrap.cc:654] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.753674 7942 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:11.753767 7942 ts_tablet_manager.cc:1367] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:11.754189 7942 raft_consensus.cc:348] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.754280 7942 raft_consensus.cc:374] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.754316 7942 raft_consensus.cc:729] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.754402 7945 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:11.754434 7942 consensus_queue.cc:260] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.754482 7945 ts_tablet_manager.cc:1367] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:11.754698 7942 ts_tablet_manager.cc:1398] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.754966 7945 raft_consensus.cc:348] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.755065 7945 raft_consensus.cc:374] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.755100 7945 raft_consensus.cc:729] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.755226 7945 consensus_queue.cc:260] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.755436 7943 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:11.755514 7945 ts_tablet_manager.cc:1398] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:11.755668 7945 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:11.756765 7943 tablet_bootstrap.cc:654] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.757475 7945 tablet_bootstrap.cc:654] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.757905 7943 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:11.757994 7943 ts_tablet_manager.cc:1367] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:11.758460 7943 raft_consensus.cc:348] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.758581 7943 raft_consensus.cc:374] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.758648 7943 raft_consensus.cc:729] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.758786 7943 consensus_queue.cc:260] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.759088 7943 ts_tablet_manager.cc:1398] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.759290 7943 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:11.759835 7945 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:11.759929 7945 ts_tablet_manager.cc:1367] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:11.760233 7943 tablet_bootstrap.cc:654] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:11.760437 7945 raft_consensus.cc:348] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.760535 7945 raft_consensus.cc:374] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.760576 7945 raft_consensus.cc:729] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.760725 7945 consensus_queue.cc:260] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.761047 7945 ts_tablet_manager.cc:1398] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:11.761576 7943 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:11.761667 7943 ts_tablet_manager.cc:1367] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:11.762121 7943 raft_consensus.cc:348] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.762233 7943 raft_consensus.cc:374] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:11.762285 7943 raft_consensus.cc:729] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:11.762408 7943 consensus_queue.cc:260] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.762689 7943 ts_tablet_manager.cc:1398] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:11.804052 7822 raft_consensus.cc:482] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.804193 7822 raft_consensus.cc:504] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.804539 7822 leader_election.cc:288] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:11.804947 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af5c8713282498abc8cf3606a7d2279" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:11.805095 7063 raft_consensus.cc:2454] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:11.805378 6869 leader_election.cc:302] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.805544 7822 raft_consensus.cc:2781] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.805619 7822 raft_consensus.cc:482] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.805652 7813 raft_consensus.cc:482] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.805696 7822 raft_consensus.cc:3037] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.805754 7813 raft_consensus.cc:504] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.806007 7813 leader_election.cc:288] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.806398 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:11.806517 7194 raft_consensus.cc:2454] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:11.806672 7822 raft_consensus.cc:504] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.806792 6868 leader_election.cc:302] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.806926 7822 leader_election.cc:288] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:11.806934 7813 raft_consensus.cc:2781] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.807008 7813 raft_consensus.cc:482] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.807055 7813 raft_consensus.cc:3037] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.807252 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af5c8713282498abc8cf3606a7d2279" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:11.807366 7063 raft_consensus.cc:3037] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.808001 7813 raft_consensus.cc:504] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.808260 7813 leader_election.cc:288] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.808398 7063 raft_consensus.cc:2454] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:11.808602 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:11.808648 6869 leader_election.cc:302] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.808717 7194 raft_consensus.cc:3037] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.808776 7813 raft_consensus.cc:2781] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.808851 7813 raft_consensus.cc:686] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:11.808962 7813 consensus_queue.cc:237] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:11.809722 7194 raft_consensus.cc:2454] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:11.810016 6868 leader_election.cc:302] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:11.810156 7813 raft_consensus.cc:2781] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.810246 7813 raft_consensus.cc:686] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:11.810367 7813 consensus_queue.cc:237] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.810397 6783 catalog_manager.cc:5484] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } }
I20240503 02:00:11.813247 6783 catalog_manager.cc:5484] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
I20240503 02:00:11.813802 7814 raft_consensus.cc:482] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:11.813879 7814 raft_consensus.cc:504] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.814138 7814 leader_election.cc:288] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.814494 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:11.814620 7194 raft_consensus.cc:2454] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 0.
I20240503 02:00:11.814898 6999 leader_election.cc:302] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:11.815057 7814 raft_consensus.cc:2781] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:11.815127 7814 raft_consensus.cc:482] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:11.815176 7814 raft_consensus.cc:3037] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.815896 7814 raft_consensus.cc:504] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.816135 7814 leader_election.cc:288] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:11.816468 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:11.816557 7194 raft_consensus.cc:3037] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:11.817332 7194 raft_consensus.cc:2454] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 1.
I20240503 02:00:11.817584 6999 leader_election.cc:302] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:11.817718 7814 raft_consensus.cc:2781] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:11.817804 7814 raft_consensus.cc:686] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:11.817915 7814 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:11.819245 6783 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
W20240503 02:00:11.935365 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:11.935580 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:11.938534 7954 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:00:12.034688 7063 raft_consensus.cc:1261] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.035091 7813 consensus_queue.cc:1035] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.038280 7194 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.038759 7888 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.044116 7195 raft_consensus.cc:1261] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:12.044690 7822 consensus_queue.cc:1035] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.99 ms
time per row: 0.056995 ms
I20240503 02:00:12.096766 6788 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:12.102151 7044 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd96db9cd364050915f5333624610a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.102355 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 701e2c3915444654989c3a817ff7b431 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.102386 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 701e2c3915444654989c3a817ff7b431 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.102461 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd96db9cd364050915f5333624610a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.102648 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701e2c3915444654989c3a817ff7b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.102655 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701e2c3915444654989c3a817ff7b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.104040 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd96db9cd364050915f5333624610a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.104285 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd96db9cd364050915f5333624610a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.104513 7942 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:12.104894 7945 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:12.105346 7942 tablet_bootstrap.cc:654] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.105578 7945 tablet_bootstrap.cc:654] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.106101 7943 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:12.106515 7942 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:12.106599 7942 ts_tablet_manager.cc:1367] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:12.106936 7943 tablet_bootstrap.cc:654] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.106994 7942 raft_consensus.cc:348] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.107101 7942 raft_consensus.cc:374] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.107131 7942 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.107260 7942 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.107522 7942 ts_tablet_manager.cc:1398] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.108114 7174 tablet_service.cc:1467] Processing CreateTablet for tablet a3abce62955548a9b7e745108b60dd1b (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.108336 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abce62955548a9b7e745108b60dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.108454 7044 tablet_service.cc:1467] Processing CreateTablet for tablet a3abce62955548a9b7e745108b60dd1b (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.108642 7943 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:12.108668 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abce62955548a9b7e745108b60dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.108711 7943 ts_tablet_manager.cc:1367] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:12.109130 7943 raft_consensus.cc:348] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.109231 7943 raft_consensus.cc:374] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.109261 7943 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.109385 7943 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.109654 7943 ts_tablet_manager.cc:1398] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.110123 7942 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:12.110289 7943 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:12.110915 7942 tablet_bootstrap.cc:654] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.111120 7943 tablet_bootstrap.cc:654] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.112361 7943 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:12.112442 7943 ts_tablet_manager.cc:1367] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.112861 7943 raft_consensus.cc:348] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.112973 7943 raft_consensus.cc:374] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.113020 7943 raft_consensus.cc:729] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.113157 7943 consensus_queue.cc:260] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.113437 7943 ts_tablet_manager.cc:1398] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.114048 7945 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:12.114130 7945 ts_tablet_manager.cc:1367] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:00:12.114241 7942 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:12.114321 7942 ts_tablet_manager.cc:1367] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:12.114576 7945 raft_consensus.cc:348] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.114674 7945 raft_consensus.cc:374] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.114715 7945 raft_consensus.cc:729] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.114692 7942 raft_consensus.cc:348] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.114801 7942 raft_consensus.cc:374] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.114836 7942 raft_consensus.cc:729] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.114843 7945 consensus_queue.cc:260] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.114953 7942 consensus_queue.cc:260] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.115078 7945 ts_tablet_manager.cc:1398] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.115216 7945 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:12.115226 7942 ts_tablet_manager.cc:1398] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:12.116109 7945 tablet_bootstrap.cc:654] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.117677 7945 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:12.117774 7945 ts_tablet_manager.cc:1367] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:12.118177 7945 raft_consensus.cc:348] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.118256 7945 raft_consensus.cc:374] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.118278 7945 raft_consensus.cc:729] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.118381 7945 consensus_queue.cc:260] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.118611 7945 ts_tablet_manager.cc:1398] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.168154 7813 raft_consensus.cc:482] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.168310 7813 raft_consensus.cc:504] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.168671 7813 leader_election.cc:288] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.169167 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:12.169330 7195 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:12.169634 6868 leader_election.cc:302] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.169790 7813 raft_consensus.cc:2781] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.169850 7813 raft_consensus.cc:482] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.169917 7813 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.170763 7813 raft_consensus.cc:504] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.170991 7813 leader_election.cc:288] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.171343 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:12.171438 7195 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.172288 7195 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:12.172619 6868 leader_election.cc:302] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.172852 7813 raft_consensus.cc:2781] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.172925 7813 raft_consensus.cc:686] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:12.173041 7813 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.174546 6790 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
I20240503 02:00:12.266916 7888 raft_consensus.cc:482] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.267041 7888 raft_consensus.cc:504] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.267356 7888 leader_election.cc:288] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.267803 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abce62955548a9b7e745108b60dd1b" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:12.267940 7195 raft_consensus.cc:2454] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 0.
I20240503 02:00:12.268250 6999 leader_election.cc:302] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:12.268438 7888 raft_consensus.cc:2781] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.268517 7888 raft_consensus.cc:482] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.268561 7888 raft_consensus.cc:3037] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.269739 7888 raft_consensus.cc:504] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.270000 7888 leader_election.cc:288] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.270370 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3abce62955548a9b7e745108b60dd1b" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:12.270478 7195 raft_consensus.cc:3037] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.271186 7195 raft_consensus.cc:2454] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 1.
I20240503 02:00:12.271451 6999 leader_election.cc:302] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:12.271595 7888 raft_consensus.cc:2781] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.271704 7888 raft_consensus.cc:686] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:12.271857 7888 consensus_queue.cc:237] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.273617 6790 catalog_manager.cc:5484] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } }
I20240503 02:00:12.374606 7813 raft_consensus.cc:482] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.374761 7813 raft_consensus.cc:504] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.375370 7813 leader_election.cc:288] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:12.375699 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd96db9cd364050915f5333624610a2" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:12.375893 7063 raft_consensus.cc:2454] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:12.376199 6869 leader_election.cc:302] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.376389 7813 raft_consensus.cc:2781] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.376467 7813 raft_consensus.cc:482] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.376499 7813 raft_consensus.cc:3037] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.377585 7813 raft_consensus.cc:504] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.377863 7813 leader_election.cc:288] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:12.378245 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd96db9cd364050915f5333624610a2" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:12.378351 7063 raft_consensus.cc:3037] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.379048 7063 raft_consensus.cc:2454] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:12.379328 6869 leader_election.cc:302] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.379520 7813 raft_consensus.cc:2781] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.379640 7813 raft_consensus.cc:686] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:12.379828 7813 consensus_queue.cc:237] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.381450 6790 catalog_manager.cc:5484] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } }
W20240503 02:00:12.502768 7990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:12.503007 7990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:12.505981 7990 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:00:12.591032 7063 raft_consensus.cc:1261] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.591106 7195 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.591485 7822 consensus_queue.cc:1035] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.591567 7813 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:12.592046 7195 raft_consensus.cc:1261] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.592495 7814 consensus_queue.cc:1035] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.7024 ms
time per row: 0.0498512 ms
I20240503 02:00:12.652369 6785 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60150:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:12.657547 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 69736194f7514beda11cce1be5711734 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.657652 7044 tablet_service.cc:1467] Processing CreateTablet for tablet 69736194f7514beda11cce1be5711734 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.657836 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69736194f7514beda11cce1be5711734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.657949 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69736194f7514beda11cce1be5711734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.658385 7174 tablet_service.cc:1467] Processing CreateTablet for tablet f40bc52bd2064bcfaa79ab4492de1320 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.658618 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bc52bd2064bcfaa79ab4492de1320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.659281 7175 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0637b4a4864fd4807a118ab4c45af3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.659487 7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0637b4a4864fd4807a118ab4c45af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.660153 8036 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:12.660774 8037 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:12.660984 8036 tablet_bootstrap.cc:654] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.661062 8035 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:12.661681 8037 tablet_bootstrap.cc:654] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.661813 7044 tablet_service.cc:1467] Processing CreateTablet for tablet 9c0637b4a4864fd4807a118ab4c45af3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.662053 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0637b4a4864fd4807a118ab4c45af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.662132 8035 tablet_bootstrap.cc:654] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.662765 8037 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:12.662897 8037 ts_tablet_manager.cc:1367] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.663130 8036 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:12.663130 8035 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:12.663214 8035 ts_tablet_manager.cc:1367] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.663214 8036 ts_tablet_manager.cc:1367] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:12.663512 8037 raft_consensus.cc:348] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.663619 8036 raft_consensus.cc:348] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.663619 8035 raft_consensus.cc:348] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.663722 8035 raft_consensus.cc:374] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.663722 8036 raft_consensus.cc:374] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.663798 8035 raft_consensus.cc:729] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.663813 8036 raft_consensus.cc:729] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.663673 8037 raft_consensus.cc:374] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.663933 8035 consensus_queue.cc:260] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.663933 8036 consensus_queue.cc:260] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.663975 8037 raft_consensus.cc:729] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.664202 8036 ts_tablet_manager.cc:1398] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.664202 8035 ts_tablet_manager.cc:1398] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.664357 8036 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Bootstrap starting.
I20240503 02:00:12.664527 6913 tablet_service.cc:1467] Processing CreateTablet for tablet f40bc52bd2064bcfaa79ab4492de1320 (DEFAULT_TABLE table=rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:12.664747 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bc52bd2064bcfaa79ab4492de1320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.665171 8036 tablet_bootstrap.cc:654] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.665654 8037 consensus_queue.cc:260] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.666000 8037 ts_tablet_manager.cc:1398] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:12.666183 8037 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de: Bootstrap starting.
I20240503 02:00:12.666213 8036 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: No bootstrap required, opened a new log
I20240503 02:00:12.666288 8036 ts_tablet_manager.cc:1367] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:12.666688 8036 raft_consensus.cc:348] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.666776 8036 raft_consensus.cc:374] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.666806 8036 raft_consensus.cc:729] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.666916 8036 consensus_queue.cc:260] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.667032 8037 tablet_bootstrap.cc:654] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.667196 8036 ts_tablet_manager.cc:1398] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.667395 8035 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: Bootstrap starting.
I20240503 02:00:12.668311 8035 tablet_bootstrap.cc:654] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:12.668857 8037 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de: No bootstrap required, opened a new log
I20240503 02:00:12.668942 8037 ts_tablet_manager.cc:1367] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:12.669423 8035 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: No bootstrap required, opened a new log
I20240503 02:00:12.669373 8037 raft_consensus.cc:348] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.669473 8037 raft_consensus.cc:374] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.669507 8035 ts_tablet_manager.cc:1367] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:12.669505 8037 raft_consensus.cc:729] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.669644 8037 consensus_queue.cc:260] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.669950 8037 ts_tablet_manager.cc:1398] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:12.670192 8035 raft_consensus.cc:348] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.670286 8035 raft_consensus.cc:374] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:12.670360 8035 raft_consensus.cc:729] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.670492 8035 consensus_queue.cc:260] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.670737 8035 ts_tablet_manager.cc:1398] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.676270 7888 raft_consensus.cc:482] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.676344 7888 raft_consensus.cc:504] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.676612 7888 leader_election.cc:288] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.676941 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:12.677040 7195 raft_consensus.cc:2454] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 0.
I20240503 02:00:12.677265 6999 leader_election.cc:302] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:12.677412 7888 raft_consensus.cc:2781] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.677464 7888 raft_consensus.cc:482] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.677508 7888 raft_consensus.cc:3037] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.678580 7888 raft_consensus.cc:504] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.678808 7888 leader_election.cc:288] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.679097 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:12.679183 7195 raft_consensus.cc:3037] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.679867 7195 raft_consensus.cc:2454] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 1.
I20240503 02:00:12.680104 6999 leader_election.cc:302] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:12.680275 7888 raft_consensus.cc:2781] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.680362 7888 raft_consensus.cc:686] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:12.680480 7888 consensus_queue.cc:237] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } }
I20240503 02:00:12.681682 6784 catalog_manager.cc:5484] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } }
I20240503 02:00:12.838945 7822 raft_consensus.cc:482] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.839063 7822 raft_consensus.cc:504] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.839349 7822 leader_election.cc:288] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.839886 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bc52bd2064bcfaa79ab4492de1320" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de" is_pre_election: true
I20240503 02:00:12.840031 7195 raft_consensus.cc:2454] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:12.840296 6868 leader_election.cc:302] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.840459 7822 raft_consensus.cc:2781] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.840521 7822 raft_consensus.cc:482] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.840569 7822 raft_consensus.cc:3037] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.841365 7822 raft_consensus.cc:504] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.841624 7822 leader_election.cc:288] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:12.841965 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40bc52bd2064bcfaa79ab4492de1320" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:12.842096 7195 raft_consensus.cc:3037] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.842805 7195 raft_consensus.cc:2454] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:12.843061 6868 leader_election.cc:302] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.843196 7822 raft_consensus.cc:2781] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.843281 7822 raft_consensus.cc:686] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:12.843413 7822 consensus_queue.cc:237] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } }
I20240503 02:00:12.844892 6784 catalog_manager.cc:5484] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } }
I20240503 02:00:12.876287 7822 raft_consensus.cc:482] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:12.876411 7822 raft_consensus.cc:504] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.876747 7822 leader_election.cc:288] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:12.877223 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69736194f7514beda11cce1be5711734" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" is_pre_election: true
I20240503 02:00:12.877358 7063 raft_consensus.cc:2454] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 0.
I20240503 02:00:12.877651 6869 leader_election.cc:302] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.877825 7822 raft_consensus.cc:2781] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:12.877877 7822 raft_consensus.cc:482] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:12.877929 7822 raft_consensus.cc:3037] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.878692 7822 raft_consensus.cc:504] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.878938 7822 leader_election.cc:288] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:12.879370 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69736194f7514beda11cce1be5711734" candidate_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:12.879473 7063 raft_consensus.cc:3037] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:12.880317 7063 raft_consensus.cc:2454] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa3c52a3f9b4bbda3ca9743ed432a08 in term 1.
I20240503 02:00:12.880643 6869 leader_election.cc:302] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:12.880815 7822 raft_consensus.cc:2781] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:12.880905 7822 raft_consensus.cc:686] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Leader. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:12.881042 7822 consensus_queue.cc:237] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } }
I20240503 02:00:12.882524 6784 catalog_manager.cc:5484] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } }
W20240503 02:00:13.002600 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:13.002821 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:13.005786 8047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:00:13.080152 7063 raft_consensus.cc:1261] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.080561 7822 consensus_queue.cc:1035] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.085137 7195 raft_consensus.cc:1261] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.085722 7813 consensus_queue.cc:1035] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:13.086916 7195 raft_consensus.cc:1261] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:13.087373 7888 consensus_queue.cc:1035] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 84.2198 ms
time per row: 0.0421099 ms
I20240503 02:00:13.129405 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:41379
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:36947
--webserver_port=44797
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:13.236408 8077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:13.236660 8077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:13.236737 8077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:13.239779 8077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:13.239869 8077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:13.239969 8077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:13.243503 8077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:41379
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=44797
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.8077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:13.244431 8077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:13.245543 8077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:13.254493 8085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.255291 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:13.256122 8077 server_base.cc:1034] running on GCE node
W20240503 02:00:13.256846 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:13.257195 8077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:13.257751 8077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:13.258888 8077 hybrid_clock.cc:648] HybridClock initialized: now 1714701613258868 us; error 41 us; skew 500 ppm
I20240503 02:00:13.260676 8077 webserver.cc:460] Webserver started at http://127.0.99.132:44797/ using document root <none> and password file <none>
I20240503 02:00:13.261219 8077 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:13.261297 8077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:13.264537 8077 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:00:13.266299 8091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.267381 8077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:00:13.267484 8077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "07bf298e40944e249d6ed0d50087aff7"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:13.267915 8077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:13.290555 8077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:13.291260 8077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:13.291419 8077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:13.292057 8077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:13.292938 8077 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:13.292984 8077 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.293027 8077 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:13.293061 8077 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.301067 8077 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41379
I20240503 02:00:13.301110 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41379 every 8 connection(s)
I20240503 02:00:13.302022 8077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:13.304823 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 8077
I20240503 02:00:13.305188 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:35021
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:36947
--webserver_port=40743
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:13.311123 8205 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:13.311334 8205 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:13.311749 8205 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:13.312809 6790 ts_manager.cc:194] Re-registered known tserver with Master: 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379)
I20240503 02:00:13.313354 6790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:55175
W20240503 02:00:13.411844 8208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:13.412040 8208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:13.412109 8208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:13.415059 8208 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:13.415123 8208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:13.415208 8208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:13.418885 8208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:35021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40743
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.8208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:13.419842 8208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:13.420934 8208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:13.429255 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.430002 8214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.430764 8216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:13.431515 8208 server_base.cc:1034] running on GCE node
I20240503 02:00:13.431912 8208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:13.432412 8208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:13.433562 8208 hybrid_clock.cc:648] HybridClock initialized: now 1714701613433539 us; error 37 us; skew 500 ppm
I20240503 02:00:13.435470 8208 webserver.cc:460] Webserver started at http://127.0.99.133:40743/ using document root <none> and password file <none>
I20240503 02:00:13.436018 8208 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:13.436085 8208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:13.439126 8208 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:13.440987 8222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.441926 8208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.442035 8208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
format_stamp: "Formatted at 2024-05-03 02:00:05 on dist-test-slave-fvr5"
I20240503 02:00:13.442422 8208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:13.459641 8208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:13.460268 8208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:13.460431 8208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:13.460966 8208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:13.462016 8229 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 02:00:13.465147 8208 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 02:00:13.465209 8208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 02:00:13.465283 8208 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 02:00:13.466679 8208 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 02:00:13.466723 8208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.466955 8229 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap starting.
I20240503 02:00:13.476662 8208 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:35021
I20240503 02:00:13.476961 8336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:35021 every 8 connection(s)
I20240503 02:00:13.477778 8208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:13.480892 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 8208
I20240503 02:00:13.481339 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:45311
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:36947
--webserver_port=38943
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:13.481675 8229 log.cc:824] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:13.484005 8229 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:00:13.484532 8229 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap complete.
I20240503 02:00:13.485971 8229 ts_tablet_manager.cc:1367] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20240503 02:00:13.486858 8337 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:13.487090 8337 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:13.487557 8337 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:13.489068 6790 ts_manager.cc:194] Re-registered known tserver with Master: 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
I20240503 02:00:13.489789 6790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:58105
I20240503 02:00:13.490115 8229 raft_consensus.cc:348] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:13.490733 8229 raft_consensus.cc:729] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.491240 8229 consensus_queue.cc:260] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:13.491842 8337 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:13.492159 8229 ts_tablet_manager.cc:1398] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20240503 02:00:13.590440 8341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:13.590659 8341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:13.590709 8341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:13.593760 8341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:13.593828 8341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:13.593894 8341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:13.597530 8341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:45311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38943
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.8341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:13.598390 8341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:13.599450 8341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:13.608088 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.608762 8347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.609611 8348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:13.610386 8341 server_base.cc:1034] running on GCE node
I20240503 02:00:13.610781 8341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:13.611347 8341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:13.612533 8341 hybrid_clock.cc:648] HybridClock initialized: now 1714701613612511 us; error 36 us; skew 500 ppm
I20240503 02:00:13.614665 8341 webserver.cc:460] Webserver started at http://127.0.99.134:38943/ using document root <none> and password file <none>
I20240503 02:00:13.615240 8341 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:13.615325 8341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:13.619884 8341 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:13.624114 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.625308 8341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:13.625419 8341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "591f812606bf438fbb9bd0a977c678e6"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:13.625891 8341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:13.639600 8341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:13.640255 8341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:13.640416 8341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:13.640960 8341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:13.641835 8341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:13.641896 8341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.641953 8341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:13.641988 8341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.650543 8341 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:45311
I20240503 02:00:13.650580 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:45311 every 8 connection(s)
I20240503 02:00:13.651546 8341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:13.657127 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 8341
I20240503 02:00:13.657480 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41723
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:36947
--webserver_port=42375
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:13.660095 8470 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:13.660300 8470 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:13.660715 8470 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:13.661613 6790 ts_manager.cc:194] Re-registered known tserver with Master: 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311)
I20240503 02:00:13.662142 6790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:36035
W20240503 02:00:13.728821 8338 tablet.cc:2369] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:13.765537 8473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:13.765769 8473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:13.765846 8473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:13.768862 8473 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:13.768931 8473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:13.769027 8473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:13.772743 8473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=42375
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.8473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:13.773584 8473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:13.774675 8473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:13.782855 8481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.783645 8478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:13.784420 8479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:13.785287 8473 server_base.cc:1034] running on GCE node
I20240503 02:00:13.785663 8473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:13.786192 8473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:13.787338 8473 hybrid_clock.cc:648] HybridClock initialized: now 1714701613787328 us; error 23 us; skew 500 ppm
I20240503 02:00:13.789176 8473 webserver.cc:460] Webserver started at http://127.0.99.135:42375/ using document root <none> and password file <none>
I20240503 02:00:13.789696 8473 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:13.789765 8473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:13.793063 8473 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:13.795058 8487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:13.796093 8473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:13.796201 8473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "12549acb31c04923a95111409fc4bad7"
format_stamp: "Formatted at 2024-05-03 02:00:06 on dist-test-slave-fvr5"
I20240503 02:00:13.796602 8473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:13.820595 8473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:13.821269 8473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:13.821430 8473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:13.821985 8473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:13.823076 8494 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 02:00:13.826100 8473 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 02:00:13.826155 8473 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 02:00:13.826234 8473 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 02:00:13.827668 8473 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 02:00:13.827711 8473 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:13.827960 8494 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:13.838021 8473 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41723
I20240503 02:00:13.838057 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41723 every 8 connection(s)
I20240503 02:00:13.839159 8473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:13.841041 8494 log.cc:824] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:13.842376 8494 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:13.842648 8494 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Bootstrap complete.
I20240503 02:00:13.843293 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 8473
I20240503 02:00:13.843461 398 test_util.cc:268] Using random seed: 345424901
I20240503 02:00:13.843847 8494 ts_tablet_manager.cc:1367] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.003s
I20240503 02:00:13.847151 8494 raft_consensus.cc:348] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:13.847575 8494 raft_consensus.cc:729] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.848085 8494 consensus_queue.cc:260] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:13.848894 8494 ts_tablet_manager.cc:1398] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:13.850266 8602 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:13.850481 8602 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:13.850970 8602 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:13.852768 6785 ts_manager.cc:194] Re-registered known tserver with Master: 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:13.853518 6785 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:57287
I20240503 02:00:13.854571 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 02:00:13.854821 398 test_util.cc:268] Using random seed: 345436259
I20240503 02:00:13.855669 8602 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:13.863150 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 02:00:13.863281 398 test_util.cc:268] Using random seed: 345444720
I20240503 02:00:13.874325 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 02:00:13.874480 398 test_util.cc:268] Using random seed: 345455918
I20240503 02:00:13.885599 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 02:00:13.885749 398 test_util.cc:268] Using random seed: 345467188
I20240503 02:00:13.915491 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 02:00:14.090265 8603 tablet.cc:2369] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:14.315116 8205 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:14.371958 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:14.377818 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.393406 6913 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:14.403829 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.404023 8271 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.404027 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:14.418617 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.624759 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.630947 8271 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.636610 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.650403 6913 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:14.651357 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:14.660355 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:14.663926 8470 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:14.692350 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:14.815938 7061 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:14.817818 7193 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch)
I20240503 02:00:14.818203 8686 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 02:00:14.822513 8663 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:14.826113 7194 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:14.829223 6785 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index -1 to 38, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
W20240503 02:00:14.831557 6999 consensus_peers.cc:485] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311): Couldn't send request to peer 591f812606bf438fbb9bd0a977c678e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77ca27caa72e415ebed755eb8a11d601. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:14.895112 6933 consensus_queue.cc:237] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:14.896627 7063 raft_consensus.cc:1261] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240503 02:00:14.897433 7822 consensus_queue.cc:1035] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 02:00:14.899250 8679 raft_consensus.cc:2932] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index -1 to 33, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:14.900511 7063 raft_consensus.cc:2932] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:14.903486 6790 catalog_manager.cc:5484] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 33, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
W20240503 02:00:14.927274 6867 consensus_peers.cc:485] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379): Couldn't send request to peer 07bf298e40944e249d6ed0d50087aff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd96db9cd364050915f5333624610a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:14.994616 7192 consensus_queue.cc:237] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:14.998368 6933 raft_consensus.cc:1261] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 02:00:14.999287 8701 consensus_queue.cc:1035] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 02:00:15.001570 8701 raft_consensus.cc:2932] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:15.003584 6933 raft_consensus.cc:2932] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
W20240503 02:00:15.005046 7130 consensus_peers.cc:485] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de -> Peer 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723): Couldn't send request to peer 12549acb31c04923a95111409fc4bad7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d9d6747dd9146f99ec8e06ef37b0c87. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.005631 6790 catalog_manager.cc:5484] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index -1 to 48, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:15.028359 8848 raft_consensus.cc:482] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:15.028658 8848 raft_consensus.cc:504] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.030822 8848 leader_election.cc:288] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:15.054963 6933 consensus_queue.cc:237] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.056098 7061 raft_consensus.cc:1261] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240503 02:00:15.056550 8696 consensus_queue.cc:1035] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240503 02:00:15.058282 8647 raft_consensus.cc:2932] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.060696 7061 raft_consensus.cc:2932] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
W20240503 02:00:15.061196 6866 consensus_peers.cc:485] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021): Couldn't send request to peer 8ac2d67357ff41b39ea5317b0874cd4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8af5c8713282498abc8cf3606a7d2279. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.066105 6785 catalog_manager.cc:5484] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 50, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.078779 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47074508be1424f9c1686bcc6526698" candidate_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "12549acb31c04923a95111409fc4bad7" is_pre_election: true
I20240503 02:00:15.079144 8556 raft_consensus.cc:2454] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac2d67357ff41b39ea5317b0874cd4c in term 1.
I20240503 02:00:15.079563 8225 leader_election.cc:302] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12549acb31c04923a95111409fc4bad7, 8ac2d67357ff41b39ea5317b0874cd4c; no voters:
I20240503 02:00:15.083832 8848 raft_consensus.cc:2781] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:15.084019 8848 raft_consensus.cc:482] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:15.084137 8848 raft_consensus.cc:3037] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:15.087291 8848 raft_consensus.cc:504] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.088044 8556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47074508be1424f9c1686bcc6526698" candidate_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "12549acb31c04923a95111409fc4bad7"
I20240503 02:00:15.088178 8556 raft_consensus.cc:3037] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:15.090253 8556 raft_consensus.cc:2454] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac2d67357ff41b39ea5317b0874cd4c in term 2.
I20240503 02:00:15.091042 8225 leader_election.cc:302] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12549acb31c04923a95111409fc4bad7, 8ac2d67357ff41b39ea5317b0874cd4c; no voters:
I20240503 02:00:15.091310 8848 leader_election.cc:288] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [CANDIDATE]: Term 2 election: Requested vote from peers 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:15.091337 8862 raft_consensus.cc:2781] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:15.091586 8862 raft_consensus.cc:686] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 LEADER]: Becoming Leader. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Running, Role: LEADER
I20240503 02:00:15.091964 8862 consensus_queue.cc:237] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.096442 6785 catalog_manager.cc:5484] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.145360 7061 consensus_queue.cc:237] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
W20240503 02:00:15.146623 6999 consensus_peers.cc:485] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311): Couldn't send request to peer 591f812606bf438fbb9bd0a977c678e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3abce62955548a9b7e745108b60dd1b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.146623 7195 raft_consensus.cc:1261] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240503 02:00:15.147744 8855 consensus_queue.cc:1035] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20240503 02:00:15.149438 8752 raft_consensus.cc:2932] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index -1 to 42, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:15.151983 7195 raft_consensus.cc:2932] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index -1 to 42, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:15.152339 6785 catalog_manager.cc:5484] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 42, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:15.201376 6933 consensus_queue.cc:237] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:15.203192 7061 raft_consensus.cc:1261] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
W20240503 02:00:15.203706 6867 consensus_peers.cc:485] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379): Couldn't send request to peer 07bf298e40944e249d6ed0d50087aff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34c7315583b84d67853e2d5de4d390d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.204008 7822 consensus_queue.cc:1035] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 02:00:15.206369 7822 raft_consensus.cc:2932] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.208385 7061 raft_consensus.cc:2932] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.209964 6787 catalog_manager.cc:5484] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 39, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.280989 6933 consensus_queue.cc:237] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:15.284569 7195 raft_consensus.cc:1261] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
I20240503 02:00:15.287186 8689 consensus_queue.cc:1035] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 53, Time since last communication: 0.000s
W20240503 02:00:15.288806 6868 consensus_peers.cc:485] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723): Couldn't send request to peer 12549acb31c04923a95111409fc4bad7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ad4302f3e0d4e1191aff07a2f39765a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.289388 7191 raft_consensus.cc:1203] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.53->[1.54-1.55] Dedup: 1.54->[1.55-1.55]
I20240503 02:00:15.291208 8679 raft_consensus.cc:2932] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:15.292649 7195 raft_consensus.cc:2932] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index -1 to 55, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:15.295517 6787 catalog_manager.cc:5484] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index -1 to 55, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:15.318794 8913 ts_tablet_manager.cc:902] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:15.324872 8913 tablet_copy_client.cc:308] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:15.338277 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet 77ca27caa72e415ebed755eb8a11d601 from peer 591f812606bf438fbb9bd0a977c678e6 ({username='slave'} at 127.0.99.134:50981)
I20240503 02:00:15.338449 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet 77ca27caa72e415ebed755eb8a11d601 from peer 591f812606bf438fbb9bd0a977c678e6 at {username='slave'} at 127.0.99.134:50981: session id = 591f812606bf438fbb9bd0a977c678e6-77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:15.339982 7084 tablet_copy_source_session.cc:205] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.345455 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca27caa72e415ebed755eb8a11d601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.358837 8913 tablet_copy_client.cc:783] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.359040 8913 tablet_copy_client.cc:647] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.366984 6933 consensus_queue.cc:237] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.369449 7064 raft_consensus.cc:1261] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
W20240503 02:00:15.369553 6866 consensus_peers.cc:485] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021): Couldn't send request to peer 8ac2d67357ff41b39ea5317b0874cd4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69736194f7514beda11cce1be5711734. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.370050 8913 tablet_copy_client.cc:523] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.372161 8913 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap starting.
I20240503 02:00:15.375700 8890 consensus_queue.cc:1035] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240503 02:00:15.380417 8890 raft_consensus.cc:2932] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.380810 7063 raft_consensus.cc:2932] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.383437 6785 catalog_manager.cc:5484] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 49, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.391361 8935 ts_tablet_manager.cc:902] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:15.394397 8935 tablet_copy_client.cc:308] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:15.399715 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 6fd96db9cd364050915f5333624610a2 from peer 07bf298e40944e249d6ed0d50087aff7 ({username='slave'} at 127.0.99.132:57507)
I20240503 02:00:15.399899 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 6fd96db9cd364050915f5333624610a2 from peer 07bf298e40944e249d6ed0d50087aff7 at {username='slave'} at 127.0.99.132:57507: session id = 07bf298e40944e249d6ed0d50087aff7-6fd96db9cd364050915f5333624610a2
I20240503 02:00:15.401381 6953 tablet_copy_source_session.cc:205] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.401674 8913 log.cc:824] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:15.405601 8935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd96db9cd364050915f5333624610a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.418033 8935 tablet_copy_client.cc:783] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.419399 8935 tablet_copy_client.cc:647] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.423347 8935 tablet_copy_client.cc:523] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.425406 8935 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap starting.
I20240503 02:00:15.449869 8935 log.cc:824] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:15.484908 8949 ts_tablet_manager.cc:902] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Initiating tablet copy from peer a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:15.486864 8949 tablet_copy_client.cc:308] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45815
I20240503 02:00:15.492657 8913 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=56 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:00:15.493156 8913 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap complete.
I20240503 02:00:15.493495 8913 ts_tablet_manager.cc:1367] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Time spent bootstrapping tablet: real 0.122s user 0.073s sys 0.000s
I20240503 02:00:15.496119 8913 raft_consensus.cc:348] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:15.497560 8913 raft_consensus.cc:729] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.498064 8913 consensus_queue.cc:260] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:15.498940 8913 ts_tablet_manager.cc:1398] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240503 02:00:15.501905 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 591f812606bf438fbb9bd0a977c678e6-77ca27caa72e415ebed755eb8a11d601 received from {username='slave'} at 127.0.99.134:50981
I20240503 02:00:15.502012 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 591f812606bf438fbb9bd0a977c678e6-77ca27caa72e415ebed755eb8a11d601 on tablet 77ca27caa72e415ebed755eb8a11d601 with peer 591f812606bf438fbb9bd0a977c678e6
I20240503 02:00:15.508955 7214 tablet_copy_service.cc:140] P a648a1aaf5eb44faa930c687aac928de: Received BeginTabletCopySession request for tablet 1d9d6747dd9146f99ec8e06ef37b0c87 from peer 12549acb31c04923a95111409fc4bad7 ({username='slave'} at 127.0.99.135:49389)
I20240503 02:00:15.509106 7214 tablet_copy_service.cc:161] P a648a1aaf5eb44faa930c687aac928de: Beginning new tablet copy session on tablet 1d9d6747dd9146f99ec8e06ef37b0c87 from peer 12549acb31c04923a95111409fc4bad7 at {username='slave'} at 127.0.99.135:49389: session id = 12549acb31c04923a95111409fc4bad7-1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:15.510529 7214 tablet_copy_source_session.cc:205] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.512890 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9d6747dd9146f99ec8e06ef37b0c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.521602 8949 tablet_copy_client.cc:783] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.521786 8949 tablet_copy_client.cc:647] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.523792 8949 tablet_copy_client.cc:523] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.525178 8949 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:15.529307 7194 consensus_queue.cc:237] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:15.530750 7063 raft_consensus.cc:1261] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240503 02:00:15.531352 8701 consensus_queue.cc:1035] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240503 02:00:15.533638 8701 raft_consensus.cc:2932] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:15.534940 7063 raft_consensus.cc:2932] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:15.540374 6787 catalog_manager.cc:5484] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 44, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
W20240503 02:00:15.546504 7130 consensus_peers.cc:485] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de -> Peer 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311): Couldn't send request to peer 591f812606bf438fbb9bd0a977c678e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c77c0d649d2c4ae39e0ade14fbe06c9e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.547421 8913 ts_tablet_manager.cc:902] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:15.548739 8913 tablet_copy_client.cc:308] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:15.549687 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet a3abce62955548a9b7e745108b60dd1b from peer 591f812606bf438fbb9bd0a977c678e6 ({username='slave'} at 127.0.99.134:50981)
I20240503 02:00:15.549808 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet a3abce62955548a9b7e745108b60dd1b from peer 591f812606bf438fbb9bd0a977c678e6 at {username='slave'} at 127.0.99.134:50981: session id = 591f812606bf438fbb9bd0a977c678e6-a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:15.550786 7084 tablet_copy_source_session.cc:205] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.552883 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3abce62955548a9b7e745108b60dd1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.555101 8913 tablet_copy_client.cc:783] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.555261 8913 tablet_copy_client.cc:647] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.562744 8913 tablet_copy_client.cc:523] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.564081 8913 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Bootstrap starting.
I20240503 02:00:15.589556 8935 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=50 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:15.590008 8935 tablet_bootstrap.cc:492] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap complete.
I20240503 02:00:15.590376 8935 ts_tablet_manager.cc:1367] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Time spent bootstrapping tablet: real 0.165s user 0.067s sys 0.004s
I20240503 02:00:15.593073 8935 raft_consensus.cc:348] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:15.593325 8935 raft_consensus.cc:729] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bf298e40944e249d6ed0d50087aff7, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.597777 8935 consensus_queue.cc:260] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:15.598757 8935 ts_tablet_manager.cc:1398] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 02:00:15.600533 8977 ts_tablet_manager.cc:902] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:15.601256 8977 tablet_copy_client.cc:308] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:15.601998 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 07bf298e40944e249d6ed0d50087aff7-6fd96db9cd364050915f5333624610a2 received from {username='slave'} at 127.0.99.132:57507
I20240503 02:00:15.602089 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 07bf298e40944e249d6ed0d50087aff7-6fd96db9cd364050915f5333624610a2 on tablet 6fd96db9cd364050915f5333624610a2 with peer 07bf298e40944e249d6ed0d50087aff7
I20240503 02:00:15.630831 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 8af5c8713282498abc8cf3606a7d2279 from peer 8ac2d67357ff41b39ea5317b0874cd4c ({username='slave'} at 127.0.99.133:41385)
I20240503 02:00:15.630940 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 8af5c8713282498abc8cf3606a7d2279 from peer 8ac2d67357ff41b39ea5317b0874cd4c at {username='slave'} at 127.0.99.133:41385: session id = 8ac2d67357ff41b39ea5317b0874cd4c-8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:15.631951 6953 tablet_copy_source_session.cc:205] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.635867 8977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af5c8713282498abc8cf3606a7d2279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.638841 8556 raft_consensus.cc:1261] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer 8ac2d67357ff41b39ea5317b0874cd4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:15.639569 8990 consensus_queue.cc:1035] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:15.661346 6933 consensus_queue.cc:237] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:15.663528 7194 raft_consensus.cc:1261] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240503 02:00:15.663826 8977 tablet_copy_client.cc:783] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.664055 8977 tablet_copy_client.cc:647] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 1 WAL segments...
W20240503 02:00:15.664192 6867 consensus_peers.cc:485] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379): Couldn't send request to peer 07bf298e40944e249d6ed0d50087aff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f40bc52bd2064bcfaa79ab4492de1320. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.667699 8679 consensus_queue.cc:1035] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 02:00:15.673285 8647 raft_consensus.cc:2932] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.682312 6787 catalog_manager.cc:5484] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 60, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.682519 7191 raft_consensus.cc:2932] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:15.689239 8913 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=727 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:15.689517 8913 tablet_bootstrap.cc:492] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Bootstrap complete.
I20240503 02:00:15.689693 8913 ts_tablet_manager.cc:1367] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Time spent bootstrapping tablet: real 0.126s user 0.063s sys 0.000s
I20240503 02:00:15.690340 8913 raft_consensus.cc:348] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:15.690457 8913 raft_consensus.cc:729] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.690627 8913 consensus_queue.cc:260] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:15.692425 8913 ts_tablet_manager.cc:1398] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:00:15.695694 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 591f812606bf438fbb9bd0a977c678e6-a3abce62955548a9b7e745108b60dd1b received from {username='slave'} at 127.0.99.134:50981
I20240503 02:00:15.695818 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 591f812606bf438fbb9bd0a977c678e6-a3abce62955548a9b7e745108b60dd1b on tablet a3abce62955548a9b7e745108b60dd1b with peer 591f812606bf438fbb9bd0a977c678e6
I20240503 02:00:15.702617 8977 tablet_copy_client.cc:523] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.704277 8977 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap starting.
I20240503 02:00:15.745318 8935 ts_tablet_manager.cc:902] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:15.745833 8935 tablet_copy_client.cc:308] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:15.746553 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 34c7315583b84d67853e2d5de4d390d5 from peer 07bf298e40944e249d6ed0d50087aff7 ({username='slave'} at 127.0.99.132:57507)
I20240503 02:00:15.746650 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 34c7315583b84d67853e2d5de4d390d5 from peer 07bf298e40944e249d6ed0d50087aff7 at {username='slave'} at 127.0.99.132:57507: session id = 07bf298e40944e249d6ed0d50087aff7-34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:15.747639 6953 tablet_copy_source_session.cc:205] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.749075 8935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c7315583b84d67853e2d5de4d390d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.751161 8935 tablet_copy_client.cc:783] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.751315 8935 tablet_copy_client.cc:647] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.763975 9018 ts_tablet_manager.cc:902] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:15.765591 8935 tablet_copy_client.cc:523] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.773015 8935 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap starting.
I20240503 02:00:15.779981 9018 tablet_copy_client.cc:308] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:15.780486 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 69736194f7514beda11cce1be5711734 from peer 8ac2d67357ff41b39ea5317b0874cd4c ({username='slave'} at 127.0.99.133:41385)
I20240503 02:00:15.780598 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 69736194f7514beda11cce1be5711734 from peer 8ac2d67357ff41b39ea5317b0874cd4c at {username='slave'} at 127.0.99.133:41385: session id = 8ac2d67357ff41b39ea5317b0874cd4c-69736194f7514beda11cce1be5711734
I20240503 02:00:15.781553 6953 tablet_copy_source_session.cc:205] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.783084 8949 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=741 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:15.783455 8949 tablet_bootstrap.cc:492] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Bootstrap complete.
I20240503 02:00:15.783690 8949 ts_tablet_manager.cc:1367] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.259s user 0.070s sys 0.005s
I20240503 02:00:15.784355 8949 raft_consensus.cc:348] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:15.784526 8949 raft_consensus.cc:729] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.784940 9018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69736194f7514beda11cce1be5711734. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.788084 9018 tablet_copy_client.cc:783] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.788278 9018 tablet_copy_client.cc:647] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.799985 8949 consensus_queue.cc:260] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:15.800412 8949 ts_tablet_manager.cc:1398] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240503 02:00:15.795919 9022 ts_tablet_manager.cc:902] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:15.802376 9022 tablet_copy_client.cc:308] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:15.808115 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 4ad4302f3e0d4e1191aff07a2f39765a from peer 12549acb31c04923a95111409fc4bad7 ({username='slave'} at 127.0.99.135:46401)
I20240503 02:00:15.808214 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 4ad4302f3e0d4e1191aff07a2f39765a from peer 12549acb31c04923a95111409fc4bad7 at {username='slave'} at 127.0.99.135:46401: session id = 12549acb31c04923a95111409fc4bad7-4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:15.809051 6953 tablet_copy_source_session.cc:205] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:15.810540 7214 tablet_copy_service.cc:342] P a648a1aaf5eb44faa930c687aac928de: Request end of tablet copy session 12549acb31c04923a95111409fc4bad7-1d9d6747dd9146f99ec8e06ef37b0c87 received from {username='slave'} at 127.0.99.135:49389
I20240503 02:00:15.810644 7214 tablet_copy_service.cc:433] P a648a1aaf5eb44faa930c687aac928de: ending tablet copy session 12549acb31c04923a95111409fc4bad7-1d9d6747dd9146f99ec8e06ef37b0c87 on tablet 1d9d6747dd9146f99ec8e06ef37b0c87 with peer 12549acb31c04923a95111409fc4bad7
I20240503 02:00:15.813012 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad4302f3e0d4e1191aff07a2f39765a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:15.815526 9022 tablet_copy_client.cc:783] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:15.815724 9022 tablet_copy_client.cc:647] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:15.817385 9018 tablet_copy_client.cc:523] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.819356 9018 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap starting.
I20240503 02:00:15.827448 9022 tablet_copy_client.cc:523] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:15.828794 9022 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:15.838948 8424 raft_consensus.cc:1203] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.70] Dedup: 1.57->[1.58-1.70]
I20240503 02:00:15.893815 8159 raft_consensus.cc:1203] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.32->[1.33-1.64] Dedup: 1.50->[1.51-1.64]
I20240503 02:00:15.937225 8424 raft_consensus.cc:1203] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.64] Dedup: 1.55->[1.56-1.64]
I20240503 02:00:15.940316 8977 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=69 overwritten=0 applied=68 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:00:15.947189 8977 tablet_bootstrap.cc:492] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap complete.
I20240503 02:00:15.947515 8977 ts_tablet_manager.cc:1367] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent bootstrapping tablet: real 0.243s user 0.071s sys 0.011s
I20240503 02:00:15.950150 8977 raft_consensus.cc:348] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.951651 8977 raft_consensus.cc:729] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.952193 8913 ts_tablet_manager.cc:902] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Initiating tablet copy from peer a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:15.952750 8913 tablet_copy_client.cc:308] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45815
I20240503 02:00:15.952737 8977 consensus_queue.cc:260] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 1.69, Last appended by leader: 69, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.953178 6933 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.953500 8977 ts_tablet_manager.cc:1398] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20240503 02:00:15.955010 6866 consensus_peers.cc:485] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021): Couldn't send request to peer 8ac2d67357ff41b39ea5317b0874cd4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 701e2c3915444654989c3a817ff7b431. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:15.958052 7194 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 02:00:15.959444 8689 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 02:00:15.959537 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-8af5c8713282498abc8cf3606a7d2279 received from {username='slave'} at 127.0.99.133:41385
I20240503 02:00:15.959630 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-8af5c8713282498abc8cf3606a7d2279 on tablet 8af5c8713282498abc8cf3606a7d2279 with peer 8ac2d67357ff41b39ea5317b0874cd4c
I20240503 02:00:15.962960 7822 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.969877 6785 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 77, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:15.975457 8556 raft_consensus.cc:1203] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.83] Dedup: 1.56->[1.57-1.83]
I20240503 02:00:15.983186 9018 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=742 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:15.983500 9018 tablet_bootstrap.cc:492] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap complete.
I20240503 02:00:15.983722 9018 ts_tablet_manager.cc:1367] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent bootstrapping tablet: real 0.164s user 0.062s sys 0.005s
I20240503 02:00:15.985261 9018 raft_consensus.cc:348] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.985374 9018 raft_consensus.cc:729] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Initialized, Role: FOLLOWER
I20240503 02:00:15.985532 9018 consensus_queue.cc:260] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:15.985893 9018 ts_tablet_manager.cc:1398] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:15.988045 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-69736194f7514beda11cce1be5711734 received from {username='slave'} at 127.0.99.133:41385
I20240503 02:00:15.988134 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-69736194f7514beda11cce1be5711734 on tablet 69736194f7514beda11cce1be5711734 with peer 8ac2d67357ff41b39ea5317b0874cd4c
I20240503 02:00:15.986907 7195 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:16.016572 7214 tablet_copy_service.cc:140] P a648a1aaf5eb44faa930c687aac928de: Received BeginTabletCopySession request for tablet c77c0d649d2c4ae39e0ade14fbe06c9e from peer 591f812606bf438fbb9bd0a977c678e6 ({username='slave'} at 127.0.99.134:59281)
I20240503 02:00:16.016703 7214 tablet_copy_service.cc:161] P a648a1aaf5eb44faa930c687aac928de: Beginning new tablet copy session on tablet c77c0d649d2c4ae39e0ade14fbe06c9e from peer 591f812606bf438fbb9bd0a977c678e6 at {username='slave'} at 127.0.99.134:59281: session id = 591f812606bf438fbb9bd0a977c678e6-c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:16.017686 7214 tablet_copy_source_session.cc:205] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:16.018682 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77c0d649d2c4ae39e0ade14fbe06c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.020910 8913 tablet_copy_client.cc:783] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:16.021062 8913 tablet_copy_client.cc:647] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:16.025354 8913 tablet_copy_client.cc:523] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:16.026835 8935 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:16.027243 8935 tablet_bootstrap.cc:492] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap complete.
I20240503 02:00:16.027477 8935 ts_tablet_manager.cc:1367] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Time spent bootstrapping tablet: real 0.255s user 0.067s sys 0.000s
I20240503 02:00:16.028170 8935 raft_consensus.cc:348] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:16.028369 8935 raft_consensus.cc:729] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bf298e40944e249d6ed0d50087aff7, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.028595 8935 consensus_queue.cc:260] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:16.029247 8913 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Bootstrap starting.
I20240503 02:00:16.031744 8935 ts_tablet_manager.cc:1398] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:16.038998 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 07bf298e40944e249d6ed0d50087aff7-34c7315583b84d67853e2d5de4d390d5 received from {username='slave'} at 127.0.99.132:57507
I20240503 02:00:16.039083 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 07bf298e40944e249d6ed0d50087aff7-34c7315583b84d67853e2d5de4d390d5 on tablet 34c7315583b84d67853e2d5de4d390d5 with peer 07bf298e40944e249d6ed0d50087aff7
I20240503 02:00:16.070102 9022 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:16.070436 9022 tablet_bootstrap.cc:492] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Bootstrap complete.
I20240503 02:00:16.070638 9022 ts_tablet_manager.cc:1367] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.242s user 0.077s sys 0.000s
I20240503 02:00:16.071120 9022 raft_consensus.cc:348] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:16.071246 9022 raft_consensus.cc:729] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.071399 9022 consensus_queue.cc:260] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:16.076900 6933 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:16.079973 8287 raft_consensus.cc:1203] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.78] Dedup: 1.69->[1.70-1.78]
I20240503 02:00:16.084857 9022 ts_tablet_manager.cc:1398] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:00:16.085486 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 12549acb31c04923a95111409fc4bad7-4ad4302f3e0d4e1191aff07a2f39765a received from {username='slave'} at 127.0.99.135:46401
I20240503 02:00:16.085569 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 12549acb31c04923a95111409fc4bad7-4ad4302f3e0d4e1191aff07a2f39765a on tablet 4ad4302f3e0d4e1191aff07a2f39765a with peer 12549acb31c04923a95111409fc4bad7
I20240503 02:00:16.093911 7063 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:00:16.094674 8679 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:00:16.100409 7822 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:16.103003 6785 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 84, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:16.107728 7060 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, VOTER 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
W20240503 02:00:16.109462 6868 consensus_peers.cc:485] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311): Couldn't send request to peer 591f812606bf438fbb9bd0a977c678e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef695f025d7c479b98b9e8b6ee4fc4e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:16.135423 8913 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:16.135923 8913 tablet_bootstrap.cc:492] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Bootstrap complete.
I20240503 02:00:16.136194 8913 ts_tablet_manager.cc:1367] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Time spent bootstrapping tablet: real 0.107s user 0.068s sys 0.004s
I20240503 02:00:16.136709 8913 raft_consensus.cc:348] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:16.136902 8913 raft_consensus.cc:729] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.137079 8913 consensus_queue.cc:260] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:16.139346 8913 ts_tablet_manager.cc:1398] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:16.141434 7214 tablet_copy_service.cc:342] P a648a1aaf5eb44faa930c687aac928de: Request end of tablet copy session 591f812606bf438fbb9bd0a977c678e6-c77c0d649d2c4ae39e0ade14fbe06c9e received from {username='slave'} at 127.0.99.134:59281
I20240503 02:00:16.141528 7214 tablet_copy_service.cc:433] P a648a1aaf5eb44faa930c687aac928de: ending tablet copy session 591f812606bf438fbb9bd0a977c678e6-c77c0d649d2c4ae39e0ade14fbe06c9e on tablet c77c0d649d2c4ae39e0ade14fbe06c9e with peer 591f812606bf438fbb9bd0a977c678e6
I20240503 02:00:16.162070 8935 ts_tablet_manager.cc:902] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:16.162575 8935 tablet_copy_client.cc:308] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:16.163640 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet f40bc52bd2064bcfaa79ab4492de1320 from peer 07bf298e40944e249d6ed0d50087aff7 ({username='slave'} at 127.0.99.132:57507)
I20240503 02:00:16.163771 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet f40bc52bd2064bcfaa79ab4492de1320 from peer 07bf298e40944e249d6ed0d50087aff7 at {username='slave'} at 127.0.99.132:57507: session id = 07bf298e40944e249d6ed0d50087aff7-f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:16.164757 6953 tablet_copy_source_session.cc:205] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:16.165737 8935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40bc52bd2064bcfaa79ab4492de1320. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.167673 8935 tablet_copy_client.cc:783] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:16.170854 8935 tablet_copy_client.cc:647] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:16.175163 8935 tablet_copy_client.cc:523] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:16.176581 8935 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap starting.
I20240503 02:00:16.181526 8287 raft_consensus.cc:1203] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.67] Dedup: 1.57->[1.58-1.67]
I20240503 02:00:16.210088 8556 raft_consensus.cc:1203] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.80] Dedup: 1.70->[1.71-1.80]
I20240503 02:00:16.211666 8159 raft_consensus.cc:1203] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.68] Dedup: 1.57->[1.58-1.68]
I20240503 02:00:16.239447 7060 consensus_queue.cc:237] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:16.244886 7191 raft_consensus.cc:1261] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240503 02:00:16.246034 8686 consensus_queue.cc:1035] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240503 02:00:16.257192 8697 raft_consensus.cc:2932] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:16.260531 7191 raft_consensus.cc:2932] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:16.261281 6787 catalog_manager.cc:5484] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 76, VOTER 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
W20240503 02:00:16.267416 6998 consensus_peers.cc:485] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 07bf298e40944e249d6ed0d50087aff7 (127.0.99.132:41379): Couldn't send request to peer 07bf298e40944e249d6ed0d50087aff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a39950d966ac43b890d417c3faa5992b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:16.292219 8935 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=702 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:16.292883 8935 tablet_bootstrap.cc:492] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Bootstrap complete.
I20240503 02:00:16.293077 8935 ts_tablet_manager.cc:1367] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Time spent bootstrapping tablet: real 0.117s user 0.067s sys 0.008s
I20240503 02:00:16.293545 8935 raft_consensus.cc:348] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:16.293673 8935 raft_consensus.cc:729] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bf298e40944e249d6ed0d50087aff7, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.293843 8935 consensus_queue.cc:260] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:16.295814 8935 ts_tablet_manager.cc:1398] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:16.298816 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 07bf298e40944e249d6ed0d50087aff7-f40bc52bd2064bcfaa79ab4492de1320 received from {username='slave'} at 127.0.99.132:57507
I20240503 02:00:16.298897 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 07bf298e40944e249d6ed0d50087aff7-f40bc52bd2064bcfaa79ab4492de1320 on tablet f40bc52bd2064bcfaa79ab4492de1320 with peer 07bf298e40944e249d6ed0d50087aff7
I20240503 02:00:16.340453 9018 ts_tablet_manager.cc:902] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:16.340948 9018 tablet_copy_client.cc:308] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:16.341318 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 701e2c3915444654989c3a817ff7b431 from peer 8ac2d67357ff41b39ea5317b0874cd4c ({username='slave'} at 127.0.99.133:41385)
I20240503 02:00:16.341408 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 701e2c3915444654989c3a817ff7b431 from peer 8ac2d67357ff41b39ea5317b0874cd4c at {username='slave'} at 127.0.99.133:41385: session id = 8ac2d67357ff41b39ea5317b0874cd4c-701e2c3915444654989c3a817ff7b431
I20240503 02:00:16.342319 6953 tablet_copy_source_session.cc:205] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:16.344692 9018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701e2c3915444654989c3a817ff7b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.346899 9018 tablet_copy_client.cc:783] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:16.347157 9018 tablet_copy_client.cc:647] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:16.354802 9018 tablet_copy_client.cc:523] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:16.356448 9018 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap starting.
I20240503 02:00:16.507668 8424 raft_consensus.cc:1203] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.80] Dedup: 1.65->[1.66-1.80]
I20240503 02:00:16.519420 9018 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=85 ignored=0} inserts{seen=716 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:00:16.519767 9018 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Bootstrap complete.
I20240503 02:00:16.519968 9018 ts_tablet_manager.cc:1367] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent bootstrapping tablet: real 0.164s user 0.083s sys 0.002s
I20240503 02:00:16.520434 9018 raft_consensus.cc:348] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:16.520931 9018 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac2d67357ff41b39ea5317b0874cd4c, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.521072 9018 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:16.521695 9018 ts_tablet_manager.cc:1398] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:16.522747 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-701e2c3915444654989c3a817ff7b431 received from {username='slave'} at 127.0.99.133:41385
I20240503 02:00:16.522827 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 8ac2d67357ff41b39ea5317b0874cd4c-701e2c3915444654989c3a817ff7b431 on tablet 701e2c3915444654989c3a817ff7b431 with peer 8ac2d67357ff41b39ea5317b0874cd4c
I20240503 02:00:16.587334 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:16.593876 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:16.602351 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:16.606211 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:16.635061 8157 raft_consensus.cc:1203] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.75] Dedup: 1.71->[1.72-1.75]
I20240503 02:00:16.653921 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:16.665853 6913 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:16.675161 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:16.752952 9162 ts_tablet_manager.cc:902] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:16.753736 9162 tablet_copy_client.cc:308] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:16.798074 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 from peer 591f812606bf438fbb9bd0a977c678e6 ({username='slave'} at 127.0.99.134:59195)
I20240503 02:00:16.798189 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet ef695f025d7c479b98b9e8b6ee4fc4e3 from peer 591f812606bf438fbb9bd0a977c678e6 at {username='slave'} at 127.0.99.134:59195: session id = 591f812606bf438fbb9bd0a977c678e6-ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:16.798899 6953 tablet_copy_source_session.cc:205] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:16.799901 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef695f025d7c479b98b9e8b6ee4fc4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.802083 9162 tablet_copy_client.cc:783] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:16.802227 9162 tablet_copy_client.cc:647] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:16.804476 9162 tablet_copy_client.cc:523] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:16.805658 9162 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap starting.
I20240503 02:00:16.869163 8291 raft_consensus.cc:1203] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.102] Dedup: 1.86->[1.87-1.102]
I20240503 02:00:16.880386 9177 ts_tablet_manager.cc:902] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:16.882688 9177 tablet_copy_client.cc:308] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:16.911105 9162 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=0 applied=96 ignored=0} inserts{seen=781 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:16.911607 9162 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Bootstrap complete.
I20240503 02:00:16.911868 9162 ts_tablet_manager.cc:1367] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Time spent bootstrapping tablet: real 0.106s user 0.087s sys 0.004s
I20240503 02:00:16.912302 9162 raft_consensus.cc:348] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:16.912485 9162 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:16.912683 9162 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 1.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:16.913426 9162 ts_tablet_manager.cc:1398] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:16.916592 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 591f812606bf438fbb9bd0a977c678e6-ef695f025d7c479b98b9e8b6ee4fc4e3 received from {username='slave'} at 127.0.99.134:59195
I20240503 02:00:16.916688 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 591f812606bf438fbb9bd0a977c678e6-ef695f025d7c479b98b9e8b6ee4fc4e3 on tablet ef695f025d7c479b98b9e8b6ee4fc4e3 with peer 591f812606bf438fbb9bd0a977c678e6
I20240503 02:00:16.943315 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet a39950d966ac43b890d417c3faa5992b from peer 07bf298e40944e249d6ed0d50087aff7 ({username='slave'} at 127.0.99.132:42553)
I20240503 02:00:16.943460 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet a39950d966ac43b890d417c3faa5992b from peer 07bf298e40944e249d6ed0d50087aff7 at {username='slave'} at 127.0.99.132:42553: session id = 07bf298e40944e249d6ed0d50087aff7-a39950d966ac43b890d417c3faa5992b
I20240503 02:00:16.944504 7084 tablet_copy_source_session.cc:205] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:16.946023 9177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39950d966ac43b890d417c3faa5992b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:16.951951 9177 tablet_copy_client.cc:783] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:16.952134 9177 tablet_copy_client.cc:647] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:16.957170 9177 tablet_copy_client.cc:523] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:16.958472 9177 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Bootstrap starting.
I20240503 02:00:17.151681 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.162475 8424 raft_consensus.cc:1203] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.83->[1.84-1.102] Dedup: 1.96->[1.97-1.102]
I20240503 02:00:17.199121 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.201349 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:17.204943 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.207464 6913 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:17.217443 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:17.236889 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:17.299304 9177 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:17.299736 9177 tablet_bootstrap.cc:492] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Bootstrap complete.
I20240503 02:00:17.299958 9177 ts_tablet_manager.cc:1367] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Time spent bootstrapping tablet: real 0.342s user 0.084s sys 0.005s
I20240503 02:00:17.300372 9177 raft_consensus.cc:348] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:17.300563 9177 raft_consensus.cc:729] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bf298e40944e249d6ed0d50087aff7, State: Initialized, Role: FOLLOWER
I20240503 02:00:17.302731 9177 consensus_queue.cc:260] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:17.303212 9177 ts_tablet_manager.cc:1398] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:17.304790 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 07bf298e40944e249d6ed0d50087aff7-a39950d966ac43b890d417c3faa5992b received from {username='slave'} at 127.0.99.132:42553
I20240503 02:00:17.304872 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 07bf298e40944e249d6ed0d50087aff7-a39950d966ac43b890d417c3faa5992b on tablet a39950d966ac43b890d417c3faa5992b with peer 07bf298e40944e249d6ed0d50087aff7
I20240503 02:00:17.368211 8158 raft_consensus.cc:1203] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.75->[1.76-1.91] Dedup: 1.86->[1.87-1.91]
I20240503 02:00:17.870123 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:17.870577 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.873262 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:17.904263 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:17.913038 6913 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:17.970352 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:17.985791 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:18.078819 6932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701e2c3915444654989c3a817ff7b431"
dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51686
I20240503 02:00:18.079001 6932 raft_consensus.cc:595] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:18.082952 8890 raft_consensus.cc:979] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: : Instructing follower a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:18.083055 8890 raft_consensus.cc:1067] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Signalling peer a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:18.091810 7193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "701e2c3915444654989c3a817ff7b431"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
from {username='slave'} at 127.0.99.129:39119
I20240503 02:00:18.091961 7193 raft_consensus.cc:482] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:18.092020 7193 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:18.093163 7193 raft_consensus.cc:504] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:18.093622 7193 leader_election.cc:288] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253), 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
I20240503 02:00:18.094699 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 137 } } ignore_live_leader: true dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:18.094820 6932 raft_consensus.cc:3032] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:18.094866 6932 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:18.095009 6932 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 137, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:18.095188 6932 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:18.096283 6932 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:18.096768 7128 leader_election.cc:302] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a648a1aaf5eb44faa930c687aac928de, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:18.097003 8906 raft_consensus.cc:2781] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:18.097081 8906 raft_consensus.cc:686] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:18.097201 8906 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 137, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:18.099449 6785 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 1 to 2, leader changed from faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:18.119153 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 137 } } ignore_live_leader: true dest_uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
I20240503 02:00:18.119390 8290 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:18.120728 8290 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
W20240503 02:00:18.178714 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41480: Illegal state: replica faa3c52a3f9b4bbda3ca9743ed432a08 is not leader of this config: current role FOLLOWER
I20240503 02:00:18.236173 6932 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 2 index: 139. (index mismatch)
I20240503 02:00:18.236689 8290 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 2 index: 139. (index mismatch)
I20240503 02:00:18.237608 8906 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240503 02:00:18.237774 8970 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240503 02:00:18.245141 9190 mvcc.cc:205] Tried to move back new op lower bound from 7023417828288651264 to 7023417827727314944. Current Snapshot: MvccSnapshot[applied={T|T < 7023417828288651264 or (T in {7023417828288651264})}]
I20240503 02:00:18.478030 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:18.496418 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:18.497826 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:18.527297 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:18.564577 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:18.566607 6913 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:18.597399 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:18.987207 7193 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 2.167, Last appended by leader: 137, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:18.988227 8290 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 2 index: 167. Preceding OpId from leader: term: 2 index: 168. (index mismatch)
I20240503 02:00:18.988668 9218 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:00:18.991729 9320 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Committing config change with OpId 2.168: config changed from index 77 to 168, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:18.992147 8291 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Committing config change with OpId 2.168: config changed from index 77 to 168, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:18.996726 6783 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 77 to 168, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:19.004376 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 701e2c3915444654989c3a817ff7b431 with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 168) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:19.004788 9375 tablet_replica.cc:330] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:19.006306 9375 raft_consensus.cc:2229] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.008098 9375 raft_consensus.cc:2258] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.009387 9375 ts_tablet_manager.cc:1875] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:19.012648 9375 ts_tablet_manager.cc:1888] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.167
I20240503 02:00:19.012799 9375 log.cc:1196] T 701e2c3915444654989c3a817ff7b431 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/701e2c3915444654989c3a817ff7b431
I20240503 02:00:19.013455 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet 701e2c3915444654989c3a817ff7b431 (table rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]) successfully deleted
I20240503 02:00:19.328438 8404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:19.373987 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:19.379266 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:19.381312 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:19.382839 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:19.386716 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:19.418435 6913 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:19.552949 7193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45210
I20240503 02:00:19.553215 7193 raft_consensus.cc:595] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Received request to transfer leadership
W20240503 02:00:19.574227 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.580735 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.589989 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.597790 7155 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.607960 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.620172 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.629694 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.643359 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.655012 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.667460 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.682374 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.694942 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.707909 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.726191 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
W20240503 02:00:19.742228 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
I20240503 02:00:19.753741 9320 raft_consensus.cc:979] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: : Instructing follower 591f812606bf438fbb9bd0a977c678e6 to start an election
I20240503 02:00:19.753872 9320 raft_consensus.cc:1067] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Signalling peer 591f812606bf438fbb9bd0a977c678e6 to start an election
I20240503 02:00:19.755858 8421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e"
dest_uuid: "591f812606bf438fbb9bd0a977c678e6"
from {username='slave'} at 127.0.99.131:33239
I20240503 02:00:19.756054 8421 raft_consensus.cc:482] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:19.756186 8421 raft_consensus.cc:3037] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:19.759816 8421 raft_consensus.cc:504] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
W20240503 02:00:19.760627 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Service unavailable: leader transfer in progress
I20240503 02:00:19.762115 8421 leader_election.cc:288] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [CANDIDATE]: Term 2 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587), a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:19.762540 7192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e" candidate_uuid: "591f812606bf438fbb9bd0a977c678e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 154 } } ignore_live_leader: true dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:19.762661 7192 raft_consensus.cc:3032] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:19.762706 7192 raft_consensus.cc:729] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:19.762856 7192 consensus_queue.cc:260] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.154, Last appended by leader: 154, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:19.763005 7192 raft_consensus.cc:3037] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:19.764582 7192 raft_consensus.cc:2454] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591f812606bf438fbb9bd0a977c678e6 in term 2.
I20240503 02:00:19.764967 8359 leader_election.cc:302] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591f812606bf438fbb9bd0a977c678e6, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:19.767318 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77c0d649d2c4ae39e0ade14fbe06c9e" candidate_uuid: "591f812606bf438fbb9bd0a977c678e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 154 } } ignore_live_leader: true dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:19.767431 7060 raft_consensus.cc:3037] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:19.768528 7060 raft_consensus.cc:2454] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591f812606bf438fbb9bd0a977c678e6 in term 2.
W20240503 02:00:19.777899 7154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59308: Illegal state: replica a648a1aaf5eb44faa930c687aac928de is not leader of this config: current role FOLLOWER
I20240503 02:00:19.777499 9446 raft_consensus.cc:2781] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:19.778162 9446 raft_consensus.cc:686] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 2 LEADER]: Becoming Leader. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Running, Role: LEADER
I20240503 02:00:19.778811 9446 consensus_queue.cc:237] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.154, Last appended by leader: 154, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:19.781344 6785 catalog_manager.cc:5484] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 reported cstate change: term changed from 1 to 2, leader changed from a648a1aaf5eb44faa930c687aac928de (127.0.99.131) to 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "591f812606bf438fbb9bd0a977c678e6" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
W20240503 02:00:19.822623 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55394: Illegal state: replica 69a1ad6834cc4c0dbd0436227b1f9dbb is not leader of this config: current role FOLLOWER
I20240503 02:00:19.828773 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:19.841380 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:19.841750 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:19.849236 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:19.854563 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:19.859864 6913 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:19.862689 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:19.866817 7195 raft_consensus.cc:1261] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 2 index: 156. (index mismatch)
I20240503 02:00:19.867305 9446 consensus_queue.cc:1035] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 02:00:19.871896 9164 mvcc.cc:205] Tried to move back new op lower bound from 7023417834969841664 to 7023417834617221120. Current Snapshot: MvccSnapshot[applied={T|T < 7023417834969841664 or (T in {7023417834969841664})}]
I20240503 02:00:19.874150 7064 raft_consensus.cc:1261] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 2 index: 156. (index mismatch)
I20240503 02:00:19.885825 9454 consensus_queue.cc:1035] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 154, Time since last communication: 0.000s
I20240503 02:00:20.229209 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:20.229467 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:20.232030 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:20.233531 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:20.243093 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:20.248461 6913 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:20.284806 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:20.451437 6933 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69736194f7514beda11cce1be5711734"
dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51738
I20240503 02:00:20.451567 6933 raft_consensus.cc:595] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:00:20.472338 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41482: Service unavailable: leader transfer in progress
W20240503 02:00:20.483426 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41482: Service unavailable: leader transfer in progress
W20240503 02:00:20.489988 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41482: Service unavailable: leader transfer in progress
I20240503 02:00:20.491130 8647 raft_consensus.cc:979] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: : Instructing follower 69a1ad6834cc4c0dbd0436227b1f9dbb to start an election
I20240503 02:00:20.491217 8647 raft_consensus.cc:1067] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Signalling peer 69a1ad6834cc4c0dbd0436227b1f9dbb to start an election
I20240503 02:00:20.496030 7064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69736194f7514beda11cce1be5711734"
dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
from {username='slave'} at 127.0.99.129:60853
I20240503 02:00:20.496212 7064 raft_consensus.cc:482] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:20.496273 7064 raft_consensus.cc:3037] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:20.497320 7064 raft_consensus.cc:504] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:20.497769 7064 leader_election.cc:288] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 2 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253), 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
W20240503 02:00:20.501302 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41482: Service unavailable: leader transfer in progress
I20240503 02:00:20.504361 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69736194f7514beda11cce1be5711734" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 167 } } ignore_live_leader: true dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:20.504482 6933 raft_consensus.cc:3032] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:20.504532 6933 raft_consensus.cc:729] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:20.504659 6933 consensus_queue.cc:260] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 167, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:20.504817 6933 raft_consensus.cc:3037] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:20.505860 6933 raft_consensus.cc:2454] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 2.
I20240503 02:00:20.507705 6997 leader_election.cc:302] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:20.508162 9321 raft_consensus.cc:2781] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:20.508260 9321 raft_consensus.cc:686] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:20.508376 9321 consensus_queue.cc:237] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 167, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:20.509507 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69736194f7514beda11cce1be5711734" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 167 } } ignore_live_leader: true dest_uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
I20240503 02:00:20.509630 8290 raft_consensus.cc:3037] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:20.510710 8290 raft_consensus.cc:2454] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 2.
W20240503 02:00:20.512085 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41482: Illegal state: replica faa3c52a3f9b4bbda3ca9743ed432a08 is not leader of this config: current role FOLLOWER
I20240503 02:00:20.517520 6783 catalog_manager.cc:5484] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 1 to 2, leader changed from faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 2 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:20.522089 8290 raft_consensus.cc:1261] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 2 index: 169. (index mismatch)
I20240503 02:00:20.522519 6932 raft_consensus.cc:1261] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 2 index: 169. (index mismatch)
I20240503 02:00:20.524050 8663 consensus_queue.cc:1035] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:00:20.524184 8697 consensus_queue.cc:1035] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240503 02:00:20.784871 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:20.812338 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:20.821534 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:20.839260 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:20.840340 6913 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:20.909796 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:20.952811 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:21.498991 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:21.520723 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:21.524691 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:21.536084 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:21.556210 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:21.564319 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:21.653376 6913 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:21.856757 6933 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a"
dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51782
I20240503 02:00:21.856917 6933 raft_consensus.cc:595] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:00:21.859418 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.867888 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.872278 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.880235 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.890298 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.903367 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.916409 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.928601 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.942157 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.970336 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:21.989416 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.002743 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.024497 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.039590 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.059432 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.079020 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.100988 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.123608 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
W20240503 02:00:22.143966 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
I20240503 02:00:22.160439 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:22.166539 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:00:22.172796 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
I20240503 02:00:22.191928 7175 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:22.198285 6913 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:22.201094 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:22.210696 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:00:22.216306 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Service unavailable: leader transfer in progress
I20240503 02:00:22.221957 7822 raft_consensus.cc:979] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: : Instructing follower a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:22.222057 7822 raft_consensus.cc:1067] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Signalling peer a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:22.222584 7191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
from {username='slave'} at 127.0.99.129:39119
I20240503 02:00:22.222777 7191 raft_consensus.cc:482] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:22.222914 7191 raft_consensus.cc:3037] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.224233 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:22.224450 7191 raft_consensus.cc:504] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:22.225883 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: true dest_uuid: "12549acb31c04923a95111409fc4bad7"
I20240503 02:00:22.226029 8554 raft_consensus.cc:3037] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.227049 8554 raft_consensus.cc:2454] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:22.227214 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: true dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:22.227350 6933 raft_consensus.cc:3032] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:22.227428 6933 raft_consensus.cc:729] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:22.227583 6933 consensus_queue.cc:260] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 236, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:22.227684 7130 leader_election.cc:302] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12549acb31c04923a95111409fc4bad7, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:22.227810 6933 raft_consensus.cc:3037] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.227955 9620 raft_consensus.cc:2781] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.228029 9620 raft_consensus.cc:686] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:22.228135 9620 consensus_queue.cc:237] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 236, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:22.228822 6933 raft_consensus.cc:2454] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:22.229187 7191 leader_election.cc:288] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253), 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:22.229825 6790 catalog_manager.cc:5484] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 1 to 2, leader changed from faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
W20240503 02:00:22.240971 6890 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41470: Illegal state: replica faa3c52a3f9b4bbda3ca9743ed432a08 is not leader of this config: current role FOLLOWER
I20240503 02:00:22.325774 8554 raft_consensus.cc:1261] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240503 02:00:22.326592 9620 consensus_queue.cc:1035] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 02:00:22.327365 6932 raft_consensus.cc:1261] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240503 02:00:22.328305 9620 consensus_queue.cc:1035] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240503 02:00:22.332007 9000 mvcc.cc:205] Tried to move back new op lower bound from 7023417845042470912 to 7023417844647505920. Current Snapshot: MvccSnapshot[applied={T|T < 7023417845042470912}]
I20240503 02:00:22.460202 7190 consensus_queue.cc:237] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 2.242, Last appended by leader: 236, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:22.461709 8556 raft_consensus.cc:1261] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 2 index: 242. Preceding OpId from leader: term: 2 index: 243. (index mismatch)
I20240503 02:00:22.462385 9320 consensus_queue.cc:1035] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s
I20240503 02:00:22.464148 9620 raft_consensus.cc:2932] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Committing config change with OpId 2.243: config changed from index 55 to 243, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:22.466943 8556 raft_consensus.cc:2932] T 4ad4302f3e0d4e1191aff07a2f39765a P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Committing config change with OpId 2.243: config changed from index 55 to 243, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:22.468469 6790 catalog_manager.cc:5484] T 4ad4302f3e0d4e1191aff07a2f39765a P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 55 to 243, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:22.470949 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad4302f3e0d4e1191aff07a2f39765a with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 243) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:22.484413 9700 tablet_replica.cc:330] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:22.484722 9700 raft_consensus.cc:2229] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:22.484930 9700 pending_rounds.cc:62] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Trying to abort 1 pending ops.
I20240503 02:00:22.485059 9700 pending_rounds.cc:68] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Aborting op as it isn't in flight: id { term: 2 index: 242 } timestamp: 7023417845564268544 op_type: WRITE_OP write_request { tablet_id: "4ad4302f3e0d4e1191aff07a2f39765a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\347\006d<^i\306@\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""f5e0da1afa1e3ad7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417845532569600 authz_token { token_data: "\010\331\214\321\261\006\"3\n\005slave\022*\n fb3785aaf951425289547f6daa48f30c\020\001\030\001 \001(\001" signature: "<redacted>""]\334n\004\367\t\257:#\274\252j^\331K\2358g%MV\255]\201p-\224\017\210\022W\007\242\371\3477\304\013f\211\376\330w\223l\326*\nm\t\0363Y\3374\300\254r\'\257\227\326_\201" signing_key_seq_num: 0 } } request_id { client_id: "8a09c9821f4e48889d8d7eca0d6d8007" seq_no: 680 first_incomplete_seq_no: 680 attempt_no: 0 }
I20240503 02:00:22.486172 9700 raft_consensus.cc:2258] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:22.487171 9700 ts_tablet_manager.cc:1875] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:22.493552 9700 ts_tablet_manager.cc:1888] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.242
I20240503 02:00:22.504272 9700 log.cc:1196] T 4ad4302f3e0d4e1191aff07a2f39765a P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:22.507997 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet 4ad4302f3e0d4e1191aff07a2f39765a (table rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]) successfully deleted
I20240503 02:00:22.893067 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:22.933943 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:22.937224 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:22.947014 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:22.950090 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:22.988250 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:23.044409 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:23.214916 7060 consensus_queue.cc:237] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 1.277, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:23.217337 8157 raft_consensus.cc:1261] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 277. Preceding OpId from leader: term: 1 index: 278. (index mismatch)
I20240503 02:00:23.217837 9321 consensus_queue.cc:1035] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20240503 02:00:23.226440 9423 raft_consensus.cc:2932] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.278: config changed from index 76 to 278, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:23.231731 8158 raft_consensus.cc:2932] T a39950d966ac43b890d417c3faa5992b P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Committing config change with OpId 1.278: config changed from index 76 to 278, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:23.233347 6790 catalog_manager.cc:5484] T a39950d966ac43b890d417c3faa5992b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index 76 to 278, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:23.289187 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet a39950d966ac43b890d417c3faa5992b with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 278) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:23.289680 9785 tablet_replica.cc:330] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:23.290071 9785 raft_consensus.cc:2229] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:23.290308 9785 raft_consensus.cc:2258] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:23.294348 9785 ts_tablet_manager.cc:1875] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:23.337198 9785 ts_tablet_manager.cc:1888] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.277
I20240503 02:00:23.338331 9785 log.cc:1196] T a39950d966ac43b890d417c3faa5992b P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a39950d966ac43b890d417c3faa5992b
I20240503 02:00:23.340322 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet a39950d966ac43b890d417c3faa5992b (table rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]) successfully deleted
I20240503 02:00:23.614354 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:23.615955 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:23.629710 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:23.635183 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:23.646445 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:23.657445 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:23.678248 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:24.031837 6933 consensus_queue.cc:237] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 287, Committed index: 287, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:24.032999 8157 raft_consensus.cc:1261] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 287. Preceding OpId from leader: term: 1 index: 288. (index mismatch)
I20240503 02:00:24.033967 9702 consensus_queue.cc:1035] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 287, Time since last communication: 0.000s
I20240503 02:00:24.036386 9702 raft_consensus.cc:2932] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 33 to 288, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:24.037746 8157 raft_consensus.cc:2932] T 6fd96db9cd364050915f5333624610a2 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 33 to 288, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:24.039315 6783 catalog_manager.cc:5484] T 6fd96db9cd364050915f5333624610a2 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index 33 to 288, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:24.077260 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd96db9cd364050915f5333624610a2 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 288) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:24.092515 9855 tablet_replica.cc:330] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:24.094859 9855 raft_consensus.cc:2229] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.095137 9855 raft_consensus.cc:2258] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.097359 9855 ts_tablet_manager.cc:1875] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:24.145087 9855 ts_tablet_manager.cc:1888] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.287
I20240503 02:00:24.145224 9855 log.cc:1196] T 6fd96db9cd364050915f5333624610a2 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fd96db9cd364050915f5333624610a2
I20240503 02:00:24.146025 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet 6fd96db9cd364050915f5333624610a2 (table rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]) successfully deleted
I20240503 02:00:24.440079 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:24.442004 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:24.443888 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:24.508615 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:24.516642 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:24.520974 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:24.525108 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:24.731004 7062 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:24.732086 8422 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240503 02:00:24.732535 8697 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240503 02:00:24.734692 8686 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 38 to 303, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:24.737694 6783 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index 38 to 303, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:24.738678 8422 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 38 to 303, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:24.775130 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ca27caa72e415ebed755eb8a11d601 with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 303) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:24.787101 9921 tablet_replica.cc:330] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:24.787369 9921 raft_consensus.cc:2229] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:24.787521 9921 raft_consensus.cc:2258] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:24.791405 9921 ts_tablet_manager.cc:1875] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:24.794770 9921 ts_tablet_manager.cc:1888] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.302
I20240503 02:00:24.794894 9921 log.cc:1196] T 77ca27caa72e415ebed755eb8a11d601 P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:24.795746 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet 77ca27caa72e415ebed755eb8a11d601 (table rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]) successfully deleted
I20240503 02:00:25.026549 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:25.031553 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:25.046254 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:25.068159 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:25.069811 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:25.086475 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:25.158919 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:25.312930 6932 consensus_queue.cc:237] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:25.315069 8156 raft_consensus.cc:1261] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 282. (index mismatch)
I20240503 02:00:25.315927 9701 consensus_queue.cc:1035] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 280, Time since last communication: 0.000s
I20240503 02:00:25.318953 9702 raft_consensus.cc:2932] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 60 to 282, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:25.324641 8157 raft_consensus.cc:2932] T f40bc52bd2064bcfaa79ab4492de1320 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 60 to 282, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:25.335076 6786 catalog_manager.cc:5484] T f40bc52bd2064bcfaa79ab4492de1320 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index 60 to 282, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:25.406013 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet f40bc52bd2064bcfaa79ab4492de1320 with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 282) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:25.406515 9993 tablet_replica.cc:330] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:25.406679 9993 raft_consensus.cc:2229] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:25.406816 9993 raft_consensus.cc:2258] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:25.407527 9993 ts_tablet_manager.cc:1875] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:25.413888 9993 ts_tablet_manager.cc:1888] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.280
I20240503 02:00:25.413973 9993 log.cc:1196] T f40bc52bd2064bcfaa79ab4492de1320 P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:25.414994 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet f40bc52bd2064bcfaa79ab4492de1320 (table rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]) successfully deleted
I20240503 02:00:25.623744 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:25.626866 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:25.650319 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:25.656292 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:25.662997 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:25.738088 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:25.762579 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:26.033582 7063 consensus_queue.cc:237] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:26.035924 8423 raft_consensus.cc:1261] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch)
I20240503 02:00:26.043375 9225 consensus_queue.cc:1035] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s
I20240503 02:00:26.049196 8686 raft_consensus.cc:2932] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index 42 to 344, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:26.052281 8423 raft_consensus.cc:2932] T a3abce62955548a9b7e745108b60dd1b P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 42 to 344, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:26.059070 6786 catalog_manager.cc:5484] T a3abce62955548a9b7e745108b60dd1b P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index 42 to 344, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:26.066476 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet a3abce62955548a9b7e745108b60dd1b with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 344) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:26.068601 10060 tablet_replica.cc:330] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:26.068806 10060 raft_consensus.cc:2229] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.070813 10060 raft_consensus.cc:2258] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.071503 10060 ts_tablet_manager.cc:1875] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:26.085692 10060 ts_tablet_manager.cc:1888] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.343
I20240503 02:00:26.085863 10060 log.cc:1196] T a3abce62955548a9b7e745108b60dd1b P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:26.086778 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet a3abce62955548a9b7e745108b60dd1b (table rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]) successfully deleted
I20240503 02:00:26.319307 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:26.321161 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:26.371327 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:26.376956 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:26.391019 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:26.444389 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:26.452939 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:26.746579 7191 consensus_queue.cc:237] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 1.408, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:26.750851 8554 raft_consensus.cc:1261] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 408. Preceding OpId from leader: term: 1 index: 409. (index mismatch)
I20240503 02:00:26.751315 9620 consensus_queue.cc:1035] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 409, Last known committed idx: 408, Time since last communication: 0.000s
I20240503 02:00:26.754195 9620 raft_consensus.cc:2932] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de [term 1 LEADER]: Committing config change with OpId 1.409: config changed from index 48 to 409, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:26.756790 6790 catalog_manager.cc:5484] T 1d9d6747dd9146f99ec8e06ef37b0c87 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 48 to 409, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:26.771373 8554 raft_consensus.cc:2932] T 1d9d6747dd9146f99ec8e06ef37b0c87 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Committing config change with OpId 1.409: config changed from index 48 to 409, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 409 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:26.792717 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9d6747dd9146f99ec8e06ef37b0c87 with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 409) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:26.793231 10132 tablet_replica.cc:330] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:26.793448 10132 raft_consensus.cc:2229] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:26.793610 10132 pending_rounds.cc:62] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Trying to abort 1 pending ops.
I20240503 02:00:26.793699 10132 pending_rounds.cc:68] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Aborting op as it isn't in flight: id { term: 1 index: 408 } timestamp: 7023417863126478848 op_type: WRITE_OP write_request { tablet_id: "1d9d6747dd9146f99ec8e06ef37b0c87" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000V\\\321u\307\304fx\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0fc25899408d59c8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417863087067136 authz_token { token_data: "\010\331\214\321\261\006\"3\n\005slave\022*\n 2df0e04b6b604b75a951a82c726cdd13\020\001\030\001 \001(\001" signature: "<redacted>""g\006$\324\266OS\"Gl\363\233Ed-NdQs\377\243 pt\021\026\206c5\203\254\376\022\004\277\331=\324\377\2178U\351\356\177\207\013\263\276d\211\361\027\254\005(\332 \222Gg\227\252\343" signing_key_seq_num: 0 } } request_id { client_id: "2d3bdb8c47af40b3b25076150f941929" seq_no: 1186 first_incomplete_seq_no: 1186 attempt_no: 0 }
I20240503 02:00:26.793994 10132 raft_consensus.cc:2258] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:26.795329 10132 ts_tablet_manager.cc:1875] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:26.797854 10132 ts_tablet_manager.cc:1888] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.408
I20240503 02:00:26.797996 10132 log.cc:1196] T 1d9d6747dd9146f99ec8e06ef37b0c87 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:26.798625 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet 1d9d6747dd9146f99ec8e06ef37b0c87 (table rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]) successfully deleted
I20240503 02:00:26.948894 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:26.952143 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:26.966313 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:26.972559 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:27.029513 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:27.030462 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:27.033635 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:27.226908 6932 consensus_queue.cc:237] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:27.228425 8291 raft_consensus.cc:1261] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch)
I20240503 02:00:27.229265 9701 consensus_queue.cc:1035] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s
I20240503 02:00:27.232087 8647 raft_consensus.cc:2932] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 50 to 363, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:27.235999 6785 catalog_manager.cc:5484] T 8af5c8713282498abc8cf3606a7d2279 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index 50 to 363, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:27.237743 8291 raft_consensus.cc:2932] T 8af5c8713282498abc8cf3606a7d2279 P 8ac2d67357ff41b39ea5317b0874cd4c [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 50 to 363, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:27.239689 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af5c8713282498abc8cf3606a7d2279 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 363) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:27.260782 10192 tablet_replica.cc:330] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:27.261015 10192 raft_consensus.cc:2229] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.261193 10192 raft_consensus.cc:2258] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.262490 10192 ts_tablet_manager.cc:1875] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:27.276469 10192 ts_tablet_manager.cc:1888] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.362
I20240503 02:00:27.279623 10192 log.cc:1196] T 8af5c8713282498abc8cf3606a7d2279 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:27.284202 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet 8af5c8713282498abc8cf3606a7d2279 (table rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]) successfully deleted
I20240503 02:00:27.538614 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:27.540438 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:27.546942 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:27.561246 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:27.588989 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:27.589295 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:27.602702 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:27.870412 6932 consensus_queue.cc:237] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 438, Committed index: 438, Last appended: 1.438, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } }
I20240503 02:00:27.871395 8157 raft_consensus.cc:1261] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 438. Preceding OpId from leader: term: 1 index: 439. (index mismatch)
I20240503 02:00:27.872361 8679 consensus_queue.cc:1035] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 439, Last known committed idx: 438, Time since last communication: 0.000s
I20240503 02:00:27.874099 8647 raft_consensus.cc:2932] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.439: config changed from index 39 to 439, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:27.875723 8157 raft_consensus.cc:2932] T 34c7315583b84d67853e2d5de4d390d5 P 07bf298e40944e249d6ed0d50087aff7 [term 1 FOLLOWER]: Committing config change with OpId 1.439: config changed from index 39 to 439, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:27.877291 6785 catalog_manager.cc:5484] T 34c7315583b84d67853e2d5de4d390d5 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index 39 to 439, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 439 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "07bf298e40944e249d6ed0d50087aff7" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41379 } } }
I20240503 02:00:27.897536 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet 34c7315583b84d67853e2d5de4d390d5 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 439) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:27.898278 10258 tablet_replica.cc:330] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:27.898453 10258 raft_consensus.cc:2229] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:27.898602 10258 raft_consensus.cc:2258] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:27.901805 10258 ts_tablet_manager.cc:1875] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:27.905314 10258 ts_tablet_manager.cc:1888] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.438
I20240503 02:00:27.905464 10258 log.cc:1196] T 34c7315583b84d67853e2d5de4d390d5 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:27.907254 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet 34c7315583b84d67853e2d5de4d390d5 (table rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]) successfully deleted
I20240503 02:00:28.291923 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.340236 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.342700 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:28.348241 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:28.354179 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:28.356199 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.367403 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.459564 6932 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3"
dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35974
I20240503 02:00:28.459702 6932 raft_consensus.cc:595] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:00:28.467032 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.471256 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.476287 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.484921 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.492817 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.505471 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.516489 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.528690 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.539280 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.553712 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.572394 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.588434 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.604462 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.616883 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:00:28.622385 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.629243 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.631444 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:00:28.641300 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.652570 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:28.652833 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:28.653069 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:00:28.665472 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.681753 6913 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:00:28.685568 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.704814 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.729730 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.750980 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.775743 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.801348 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.827240 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.855582 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
W20240503 02:00:28.881996 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.904166 7822 raft_consensus.cc:979] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: : Instructing follower 69a1ad6834cc4c0dbd0436227b1f9dbb to start an election
I20240503 02:00:28.904269 7822 raft_consensus.cc:1067] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Signalling peer 69a1ad6834cc4c0dbd0436227b1f9dbb to start an election
W20240503 02:00:28.909832 6892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Service unavailable: leader transfer in progress
I20240503 02:00:28.918571 7064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3"
dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
from {username='slave'} at 127.0.99.129:60853
I20240503 02:00:28.918730 7064 raft_consensus.cc:482] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:28.918795 7064 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:28.920003 7064 raft_consensus.cc:504] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:28.920367 7064 leader_election.cc:288] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 2 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253), 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311)
I20240503 02:00:28.920800 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 434 } } ignore_live_leader: true dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:28.920920 6932 raft_consensus.cc:3032] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:28.920959 6932 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:28.921087 6932 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 1.434, Last appended by leader: 434, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:28.921222 6932 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:28.921981 8421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 434 } } ignore_live_leader: true dest_uuid: "591f812606bf438fbb9bd0a977c678e6"
I20240503 02:00:28.922133 8421 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:28.922266 6932 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 2.
I20240503 02:00:28.923213 6997 leader_election.cc:302] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, faa3c52a3f9b4bbda3ca9743ed432a08; no voters:
I20240503 02:00:28.923254 8421 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a1ad6834cc4c0dbd0436227b1f9dbb in term 2.
I20240503 02:00:28.923556 10248 raft_consensus.cc:2781] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:28.923663 10248 raft_consensus.cc:686] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Becoming Leader. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:28.923815 10248 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 1.434, Last appended by leader: 434, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:28.930706 6790 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: term changed from 1 to 2, leader changed from faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) to 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130). New cstate: current_term: 2 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
W20240503 02:00:28.940066 6893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41462: Illegal state: replica faa3c52a3f9b4bbda3ca9743ed432a08 is not leader of this config: current role FOLLOWER
I20240503 02:00:28.973569 8421 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 2 index: 436. (index mismatch)
I20240503 02:00:28.974226 8686 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 02:00:28.975061 6932 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 2 index: 436. (index mismatch)
I20240503 02:00:28.977809 9567 mvcc.cc:205] Tried to move back new op lower bound from 7023417872272101376 to 7023417872073392128. Current Snapshot: MvccSnapshot[applied={T|T < 7023417872272101376 or (T in {7023417872272101376})}]
I20240503 02:00:28.980399 10204 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240503 02:00:28.997682 9838 mvcc.cc:205] Tried to move back new op lower bound from 7023417872272101376 to 7023417872073392128. Current Snapshot: MvccSnapshot[applied={T|T < 7023417872272101376}]
I20240503 02:00:29.049260 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:29.062094 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:29.064704 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:29.075377 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:29.081975 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:29.082310 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:29.192656 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:29.403606 8424 consensus_queue.cc:237] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 2.496, Last appended by leader: 154, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:29.404757 7064 raft_consensus.cc:1261] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 496. Preceding OpId from leader: term: 2 index: 497. (index mismatch)
I20240503 02:00:29.405150 10259 consensus_queue.cc:1035] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240503 02:00:29.407909 10259 raft_consensus.cc:2932] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 [term 2 LEADER]: Committing config change with OpId 2.497: config changed from index 44 to 497, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:29.410959 6784 catalog_manager.cc:5484] T c77c0d649d2c4ae39e0ade14fbe06c9e P 591f812606bf438fbb9bd0a977c678e6 reported cstate change: config changed from index 44 to 497, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "591f812606bf438fbb9bd0a977c678e6" committed_config { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:29.413233 7064 raft_consensus.cc:2932] T c77c0d649d2c4ae39e0ade14fbe06c9e P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Committing config change with OpId 2.497: config changed from index 44 to 497, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:29.419430 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet c77c0d649d2c4ae39e0ade14fbe06c9e with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 497) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:29.422025 10422 tablet_replica.cc:330] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:29.422189 10422 raft_consensus.cc:2229] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:29.422309 10422 raft_consensus.cc:2258] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:29.423931 10422 ts_tablet_manager.cc:1875] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:29.427588 10422 ts_tablet_manager.cc:1888] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.496
I20240503 02:00:29.427739 10422 log.cc:1196] T c77c0d649d2c4ae39e0ade14fbe06c9e P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:29.429636 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet c77c0d649d2c4ae39e0ade14fbe06c9e (table rebalance_test_table_0 [id=2df0e04b6b604b75a951a82c726cdd13]) successfully deleted
I20240503 02:00:29.714346 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:29.725389 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:29.738993 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:29.740796 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:29.760185 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:29.763283 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:29.792559 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:30.009905 7062 consensus_queue.cc:237] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 2.424, Last appended by leader: 167, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } }
I20240503 02:00:30.012109 8290 raft_consensus.cc:1261] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 424. Preceding OpId from leader: term: 2 index: 425. (index mismatch)
I20240503 02:00:30.012629 10386 consensus_queue.cc:1035] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 425, Last known committed idx: 424, Time since last communication: 0.000s
I20240503 02:00:30.014744 10248 raft_consensus.cc:2932] T 69736194f7514beda11cce1be5711734 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Committing config change with OpId 2.425: config changed from index 49 to 425, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:30.017063 8290 raft_consensus.cc:2932] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Committing config change with OpId 2.425: config changed from index 49 to 425, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:30.019927 6785 catalog_manager.cc:5484] T 69736194f7514beda11cce1be5711734 P 8ac2d67357ff41b39ea5317b0874cd4c reported cstate change: config changed from index 49 to 425, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 425 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } }
I20240503 02:00:30.025837 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 69736194f7514beda11cce1be5711734 with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 425) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:30.026218 10492 tablet_replica.cc:330] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:30.026489 10492 raft_consensus.cc:2229] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.026651 10492 raft_consensus.cc:2258] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.027444 10492 ts_tablet_manager.cc:1875] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.030198 10492 ts_tablet_manager.cc:1888] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.424
I20240503 02:00:30.030335 10492 log.cc:1196] T 69736194f7514beda11cce1be5711734 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/69736194f7514beda11cce1be5711734
I20240503 02:00:30.031630 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet 69736194f7514beda11cce1be5711734 (table rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]) successfully deleted
I20240503 02:00:30.460651 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:30.464547 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:30.476581 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:30.485494 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:30.487226 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:30.507094 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:30.507702 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:30.780071 7064 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 2.502, Last appended by leader: 434, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } }
I20240503 02:00:30.781709 8423 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 502. Preceding OpId from leader: term: 2 index: 503. (index mismatch)
I20240503 02:00:30.782281 8686 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 501, Time since last communication: 0.000s
I20240503 02:00:30.786684 8663 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Committing config change with OpId 2.503: config changed from index 84 to 503, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:30.790197 8424 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Committing config change with OpId 2.503: config changed from index 84 to 503, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:30.791158 6785 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index 84 to 503, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } }
I20240503 02:00:30.798252 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 503) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:30.799036 10606 tablet_replica.cc:330] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:30.799677 10606 raft_consensus.cc:2229] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:30.799844 10606 raft_consensus.cc:2258] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:30.801440 10606 ts_tablet_manager.cc:1875] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:30.807832 10606 ts_tablet_manager.cc:1888] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.502
I20240503 02:00:30.809108 10606 log.cc:1196] T ef695f025d7c479b98b9e8b6ee4fc4e3 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:30.812681 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet ef695f025d7c479b98b9e8b6ee4fc4e3 (table rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]) successfully deleted
I20240503 02:00:31.079771 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:31.088657 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:31.110946 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:31.124382 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:31.153443 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:31.154580 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:31.181593 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:31.391953 7064 consensus_queue.cc:237] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 484, Committed index: 484, Last appended: 1.484, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:31.395213 7195 raft_consensus.cc:1261] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 484. Preceding OpId from leader: term: 1 index: 485. (index mismatch)
I20240503 02:00:31.402283 10248 consensus_queue.cc:1035] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 485, Last known committed idx: 484, Time since last communication: 0.000s
I20240503 02:00:31.405195 8663 raft_consensus.cc:2932] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.485: config changed from index -1 to 485, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:31.407665 7195 raft_consensus.cc:2932] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.485: config changed from index -1 to 485, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:31.410362 6790 catalog_manager.cc:5484] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index -1 to 485, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
W20240503 02:00:31.419435 6999 consensus_peers.cc:485] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723): Couldn't send request to peer 12549acb31c04923a95111409fc4bad7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c0637b4a4864fd4807a118ab4c45af3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:31.487110 7064 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 2.528, Last appended by leader: 434, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:31.488651 8424 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 528. Preceding OpId from leader: term: 2 index: 529. (index mismatch)
W20240503 02:00:31.489061 6999 consensus_peers.cc:485] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723): Couldn't send request to peer 12549acb31c04923a95111409fc4bad7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef695f025d7c479b98b9e8b6ee4fc4e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:31.489147 8663 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I20240503 02:00:31.491015 8663 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Committing config change with OpId 2.529: config changed from index 503 to 529, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:31.492115 8424 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Committing config change with OpId 2.529: config changed from index 503 to 529, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:31.495440 6784 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 reported cstate change: config changed from index 503 to 529, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:31.811691 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:31.819175 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:31.826099 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:31.826308 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:31.834208 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:31.842314 10746 ts_tablet_manager.cc:902] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:31.844892 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:31.862798 10746 tablet_copy_client.cc:308] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:31.881510 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet 9c0637b4a4864fd4807a118ab4c45af3 from peer 12549acb31c04923a95111409fc4bad7 ({username='slave'} at 127.0.99.135:33983)
I20240503 02:00:31.881654 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet 9c0637b4a4864fd4807a118ab4c45af3 from peer 12549acb31c04923a95111409fc4bad7 at {username='slave'} at 127.0.99.135:33983: session id = 12549acb31c04923a95111409fc4bad7-9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:31.882658 7084 tablet_copy_source_session.cc:205] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.892189 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c0637b4a4864fd4807a118ab4c45af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.894474 10746 tablet_copy_client.cc:783] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.894627 10746 tablet_copy_client.cc:647] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.899276 10746 tablet_copy_client.cc:523] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.900601 10746 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:31.902058 10752 ts_tablet_manager.cc:902] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:31.904593 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:31.913419 10752 tablet_copy_client.cc:308] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:31.914402 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 from peer 12549acb31c04923a95111409fc4bad7 ({username='slave'} at 127.0.99.135:33983)
I20240503 02:00:31.914515 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet ef695f025d7c479b98b9e8b6ee4fc4e3 from peer 12549acb31c04923a95111409fc4bad7 at {username='slave'} at 127.0.99.135:33983: session id = 12549acb31c04923a95111409fc4bad7-ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:31.915565 7084 tablet_copy_source_session.cc:205] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:31.921036 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef695f025d7c479b98b9e8b6ee4fc4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:31.923408 10752 tablet_copy_client.cc:783] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:31.923563 10752 tablet_copy_client.cc:647] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:31.938354 10752 tablet_copy_client.cc:523] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:31.959247 10752 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Bootstrap starting.
I20240503 02:00:32.241724 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:32.242105 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:32.246965 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:32.267165 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:32.275625 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:32.317464 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:32.332907 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:32.479454 10746 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Bootstrap replayed 1/1 log segments. Stats: ops{read=498 overwritten=0 applied=498 ignored=0} inserts{seen=1170 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:32.516773 10746 tablet_bootstrap.cc:492] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Bootstrap complete.
I20240503 02:00:32.517073 10746 ts_tablet_manager.cc:1367] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.617s user 0.363s sys 0.012s
I20240503 02:00:32.517593 10746 raft_consensus.cc:348] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:32.517711 10746 raft_consensus.cc:729] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:32.517840 10746 consensus_queue.cc:260] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 498, Last appended: 1.498, Last appended by leader: 498, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:32.518172 10746 ts_tablet_manager.cc:1398] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:32.520881 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 12549acb31c04923a95111409fc4bad7-9c0637b4a4864fd4807a118ab4c45af3 received from {username='slave'} at 127.0.99.135:33983
I20240503 02:00:32.520970 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 12549acb31c04923a95111409fc4bad7-9c0637b4a4864fd4807a118ab4c45af3 on tablet 9c0637b4a4864fd4807a118ab4c45af3 with peer 12549acb31c04923a95111409fc4bad7
I20240503 02:00:32.614580 10752 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Bootstrap replayed 1/1 log segments. Stats: ops{read=549 overwritten=0 applied=549 ignored=0} inserts{seen=1231 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:32.614918 10752 tablet_bootstrap.cc:492] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Bootstrap complete.
I20240503 02:00:32.615149 10752 ts_tablet_manager.cc:1367] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Time spent bootstrapping tablet: real 0.656s user 0.388s sys 0.021s
I20240503 02:00:32.615553 10752 raft_consensus.cc:348] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:32.615662 10752 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12549acb31c04923a95111409fc4bad7, State: Initialized, Role: FOLLOWER
I20240503 02:00:32.615794 10752 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 549, Last appended: 2.549, Last appended by leader: 549, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:32.616060 10752 ts_tablet_manager.cc:1398] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:32.619699 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 12549acb31c04923a95111409fc4bad7-ef695f025d7c479b98b9e8b6ee4fc4e3 received from {username='slave'} at 127.0.99.135:33983
I20240503 02:00:32.619907 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 12549acb31c04923a95111409fc4bad7-ef695f025d7c479b98b9e8b6ee4fc4e3 on tablet ef695f025d7c479b98b9e8b6ee4fc4e3 with peer 12549acb31c04923a95111409fc4bad7
I20240503 02:00:32.689169 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:32.689576 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:32.689872 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:32.732090 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:32.736202 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:32.745186 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:32.752074 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:32.761783 8554 raft_consensus.cc:1203] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.484->[1.485-1.514] Dedup: 1.498->[1.499-1.514]
I20240503 02:00:32.991442 8556 raft_consensus.cc:1203] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.528->[2.529-2.572] Dedup: 2.549->[2.550-2.572]
I20240503 02:00:33.061851 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:33.096812 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:33.128458 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:33.130331 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:33.143586 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:33.161751 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:33.174119 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:33.641263 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:33.677115 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:33.687376 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:33.687649 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:33.710927 8404 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:33.748880 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:33.816560 7044 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:33.890777 7062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3"
dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34730
I20240503 02:00:33.890969 7062 raft_consensus.cc:595] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Received request to transfer leadership
I20240503 02:00:33.902730 10204 raft_consensus.cc:979] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: : Instructing follower 591f812606bf438fbb9bd0a977c678e6 to start an election
I20240503 02:00:33.902837 10204 raft_consensus.cc:1067] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Signalling peer 591f812606bf438fbb9bd0a977c678e6 to start an election
W20240503 02:00:33.920168 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52918: Service unavailable: leader transfer in progress
W20240503 02:00:33.922610 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52918: Service unavailable: leader transfer in progress
I20240503 02:00:33.924595 8422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3"
dest_uuid: "591f812606bf438fbb9bd0a977c678e6"
from {username='slave'} at 127.0.99.130:40017
I20240503 02:00:33.924762 8422 raft_consensus.cc:482] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:33.924829 8422 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:33.926401 8422 raft_consensus.cc:504] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:33.927871 8422 leader_election.cc:288] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [CANDIDATE]: Term 3 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587), 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
W20240503 02:00:33.929347 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52918: Service unavailable: leader transfer in progress
I20240503 02:00:33.931876 7062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "591f812606bf438fbb9bd0a977c678e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 606 } } ignore_live_leader: true dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:33.932000 7062 raft_consensus.cc:3032] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:33.932044 7062 raft_consensus.cc:729] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:33.932217 7062 consensus_queue.cc:260] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 606, Committed index: 606, Last appended: 2.606, Last appended by leader: 606, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:33.932397 7062 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:33.933521 7062 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591f812606bf438fbb9bd0a977c678e6 in term 3.
I20240503 02:00:33.935122 8360 leader_election.cc:302] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591f812606bf438fbb9bd0a977c678e6, 69a1ad6834cc4c0dbd0436227b1f9dbb; no voters:
I20240503 02:00:33.935379 10848 raft_consensus.cc:2781] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:33.935467 10848 raft_consensus.cc:686] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 3 LEADER]: Becoming Leader. State: Replica: 591f812606bf438fbb9bd0a977c678e6, State: Running, Role: LEADER
I20240503 02:00:33.935585 10848 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 606, Committed index: 606, Last appended: 2.606, Last appended by leader: 606, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:33.937412 6785 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 reported cstate change: term changed from 2 to 3, leader changed from 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) to 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134). New cstate: current_term: 3 leader_uuid: "591f812606bf438fbb9bd0a977c678e6" committed_config { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
W20240503 02:00:33.938019 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52918: Illegal state: replica 69a1ad6834cc4c0dbd0436227b1f9dbb is not leader of this config: current role FOLLOWER
W20240503 02:00:33.957576 7023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52918: Illegal state: replica 69a1ad6834cc4c0dbd0436227b1f9dbb is not leader of this config: current role FOLLOWER
I20240503 02:00:33.962621 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" candidate_uuid: "591f812606bf438fbb9bd0a977c678e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 606 } } ignore_live_leader: true dest_uuid: "12549acb31c04923a95111409fc4bad7"
I20240503 02:00:33.962771 8554 raft_consensus.cc:3037] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:33.963966 8554 raft_consensus.cc:2454] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591f812606bf438fbb9bd0a977c678e6 in term 3.
I20240503 02:00:33.980839 7062 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 3 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 606. Preceding OpId from leader: term: 3 index: 608. (index mismatch)
I20240503 02:00:33.980835 8554 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 3 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 606. Preceding OpId from leader: term: 3 index: 608. (index mismatch)
I20240503 02:00:33.981511 10848 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 607, Last known committed idx: 606, Time since last communication: 0.000s
I20240503 02:00:33.983538 10848 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 607, Last known committed idx: 606, Time since last communication: 0.000s
I20240503 02:00:34.104300 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:34.109405 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:34.144776 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:34.150936 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:34.162662 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:34.164760 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:34.197866 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:34.525972 8422 consensus_queue.cc:237] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 623, Committed index: 623, Last appended: 3.624, Last appended by leader: 606, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:34.528556 8554 raft_consensus.cc:1261] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 3 FOLLOWER]: Refusing update from remote peer 591f812606bf438fbb9bd0a977c678e6: Log matching property violated. Preceding OpId in replica: term: 3 index: 623. Preceding OpId from leader: term: 3 index: 625. (index mismatch)
I20240503 02:00:34.528960 10989 consensus_queue.cc:1035] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 625, Last known committed idx: 623, Time since last communication: 0.000s
I20240503 02:00:34.531950 10848 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 [term 3 LEADER]: Committing config change with OpId 3.625: config changed from index 529 to 625, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:34.533474 8556 raft_consensus.cc:1203] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.623->[3.624-3.624] Dedup: 3.624->[]
I20240503 02:00:34.537931 6786 catalog_manager.cc:5484] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 591f812606bf438fbb9bd0a977c678e6 reported cstate change: config changed from index 529 to 625, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "591f812606bf438fbb9bd0a977c678e6" committed_config { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:34.541365 8554 raft_consensus.cc:2932] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 12549acb31c04923a95111409fc4bad7 [term 3 FOLLOWER]: Committing config change with OpId 3.625: config changed from index 529 to 625, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 625 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:34.571956 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet ef695f025d7c479b98b9e8b6ee4fc4e3 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 625) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:34.582808 11025 tablet_replica.cc:330] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:34.586398 11025 raft_consensus.cc:2229] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:34.588876 11025 pending_rounds.cc:62] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Trying to abort 1 pending ops.
I20240503 02:00:34.589049 11025 pending_rounds.cc:68] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Aborting op as it isn't in flight: id { term: 3 index: 624 } timestamp: 7023417894971973632 op_type: WRITE_OP write_request { tablet_id: "ef695f025d7c479b98b9e8b6ee4fc4e3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\252\225-M\177G\025\302\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a28b2bc03dcab7ef" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023417894939492352 authz_token { token_data: "\010\331\214\321\261\006\"3\n\005slave\022*\n 923688f0034044adbaeb3caae37e4e28\020\001\030\001 \001(\001" signature: "<redacted>""\031b\201\300fW\233C\211>\245\266p4jI\235&+\262\021\177\030\255\322\303\013\2565~h\270\336\275\2226<\004\233Fv\014\2164Z\343\335\266u\266\020\235?\032:\325\341\320\301,\340\245\311\030" signing_key_seq_num: 0 } } request_id { client_id: "b9ed643f8f0e4500a42084ebdfcc3be6" seq_no: 1840 first_incomplete_seq_no: 1840 attempt_no: 0 }
I20240503 02:00:34.591303 11025 raft_consensus.cc:2258] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:34.592523 11025 ts_tablet_manager.cc:1875] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:34.595342 11025 ts_tablet_manager.cc:1888] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.624
I20240503 02:00:34.595482 11025 log.cc:1196] T ef695f025d7c479b98b9e8b6ee4fc4e3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:34.596416 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet ef695f025d7c479b98b9e8b6ee4fc4e3 (table rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]) successfully deleted
I20240503 02:00:34.813153 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:34.828321 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:34.829850 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:34.830427 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:34.832223 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:34.839984 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:34.845654 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:34.955897 7062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3"
dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34774
I20240503 02:00:34.956048 7062 raft_consensus.cc:595] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Received request to transfer leadership
W20240503 02:00:35.021579 7024 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.025849 7024 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.031150 7024 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.038522 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.045854 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.055891 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.066401 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.081383 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.096329 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.111315 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.124047 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
I20240503 02:00:35.127426 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:35.128391 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:35.135347 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:35.137881 7175 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:00:35.140821 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
I20240503 02:00:35.143008 7044 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:35.153970 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:00:35.161130 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.177276 7024 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.194094 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
I20240503 02:00:35.194401 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:00:35.215171 7024 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.236601 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.257105 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.281105 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.306510 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
W20240503 02:00:35.331928 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Service unavailable: leader transfer in progress
I20240503 02:00:35.334697 10204 raft_consensus.cc:979] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: : Instructing follower a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:35.334789 10204 raft_consensus.cc:1067] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Signalling peer a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:35.335464 7191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
from {username='slave'} at 127.0.99.130:47177
I20240503 02:00:35.335592 7191 raft_consensus.cc:482] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:35.335652 7191 raft_consensus.cc:3037] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:35.336752 7191 raft_consensus.cc:504] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:35.337246 7191 leader_election.cc:288] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587), 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723)
I20240503 02:00:35.337467 7062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 575 } } ignore_live_leader: true dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:35.337579 7062 raft_consensus.cc:3032] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:35.337617 7062 raft_consensus.cc:729] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:35.337780 7062 consensus_queue.cc:260] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 575, Committed index: 575, Last appended: 1.575, Last appended by leader: 575, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:35.337931 7062 raft_consensus.cc:3037] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:35.338433 8555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c0637b4a4864fd4807a118ab4c45af3" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 575 } } ignore_live_leader: true dest_uuid: "12549acb31c04923a95111409fc4bad7"
I20240503 02:00:35.338575 8555 raft_consensus.cc:3037] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:35.338982 7062 raft_consensus.cc:2454] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:35.339730 7131 leader_election.cc:302] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a1ad6834cc4c0dbd0436227b1f9dbb, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:35.339936 10977 raft_consensus.cc:2781] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:35.340008 10977 raft_consensus.cc:686] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:35.340135 10977 consensus_queue.cc:237] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 575, Committed index: 575, Last appended: 1.575, Last appended by leader: 575, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:35.342104 6787 catalog_manager.cc:5484] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 1 to 2, leader changed from 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 485 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:35.345415 8555 raft_consensus.cc:2454] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
W20240503 02:00:35.357791 7021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52940: Illegal state: replica 69a1ad6834cc4c0dbd0436227b1f9dbb is not leader of this config: current role FOLLOWER
I20240503 02:00:35.391239 8555 raft_consensus.cc:1261] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 575. Preceding OpId from leader: term: 2 index: 577. (index mismatch)
I20240503 02:00:35.391583 7064 raft_consensus.cc:1261] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 575. Preceding OpId from leader: term: 2 index: 577. (index mismatch)
I20240503 02:00:35.392287 10977 consensus_queue.cc:1035] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 576, Last known committed idx: 575, Time since last communication: 0.000s
I20240503 02:00:35.392459 10643 consensus_queue.cc:1035] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 576, Last known committed idx: 575, Time since last communication: 0.000s
I20240503 02:00:35.666077 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:35.673830 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:35.687978 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:35.713606 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:35.732977 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:35.745499 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:35.746723 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:35.954205 7191 consensus_queue.cc:237] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 595, Committed index: 595, Last appended: 2.595, Last appended by leader: 575, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 596 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } }
I20240503 02:00:35.955345 8556 raft_consensus.cc:1261] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 2 index: 595. Preceding OpId from leader: term: 2 index: 596. (index mismatch)
I20240503 02:00:35.955804 9320 consensus_queue.cc:1035] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 596, Last known committed idx: 595, Time since last communication: 0.000s
I20240503 02:00:35.957816 10801 raft_consensus.cc:2932] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Committing config change with OpId 2.596: config changed from index 485 to 596, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 596 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:35.960510 6785 catalog_manager.cc:5484] T 9c0637b4a4864fd4807a118ab4c45af3 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 485 to 596, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 596 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:35.960971 8554 raft_consensus.cc:2932] T 9c0637b4a4864fd4807a118ab4c45af3 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Committing config change with OpId 2.596: config changed from index 485 to 596, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 596 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } }
I20240503 02:00:35.987100 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c0637b4a4864fd4807a118ab4c45af3 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 596) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:35.987643 11192 tablet_replica.cc:330] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:35.987871 11192 raft_consensus.cc:2229] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:35.988065 11192 raft_consensus.cc:2258] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:35.989079 11192 ts_tablet_manager.cc:1875] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:35.994374 11192 ts_tablet_manager.cc:1888] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.595
I20240503 02:00:35.994518 11192 log.cc:1196] T 9c0637b4a4864fd4807a118ab4c45af3 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:35.995211 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet 9c0637b4a4864fd4807a118ab4c45af3 (table rebalance_test_table_4 [id=55a11c577ade4bfaa6701fb5567a7d73]) successfully deleted
I20240503 02:00:36.259856 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.274664 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:36.287063 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:36.295185 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:36.305532 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:36.311722 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:36.372388 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.750214 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.790562 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:36.790926 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:36.821595 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:36.822492 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:36.833250 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:36.870752 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:37.397876 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.415144 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.432327 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.453074 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:37.475919 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:37.486733 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:37.487216 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:37.848026 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.867480 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:37.896473 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:37.901901 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:37.915035 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:37.927886 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:37.959666 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:38.151716 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:38.152851 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:38.155267 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:38.155934 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:38.161352 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:38.169829 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:38.177757 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
cc32e3225a044351a09cc29d3e9fff23 | 127.0.99.190:36947 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43091 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
07bf298e40944e249d6ed0d50087aff7 | 127.0.99.132:41379 | HEALTHY | <none> | 0 | 0
12549acb31c04923a95111409fc4bad7 | 127.0.99.135:41723 | HEALTHY | <none> | 0 | 0
591f812606bf438fbb9bd0a977c678e6 | 127.0.99.134:45311 | HEALTHY | <none> | 2 | 0
69a1ad6834cc4c0dbd0436227b1f9dbb | 127.0.99.130:46587 | HEALTHY | <none> | 4 | 0
8ac2d67357ff41b39ea5317b0874cd4c | 127.0.99.133:35021 | HEALTHY | <none> | 1 | 0
a648a1aaf5eb44faa930c687aac928de | 127.0.99.131:45815 | HEALTHY | <none> | 4 | 0
faa3c52a3f9b4bbda3ca9743ed432a08 | 127.0.99.129:45253 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:45253
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:46587
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:45815
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41379
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:35021
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:45311
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:41723
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:45253
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:46587
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:45815
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41379
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:35021
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:45311
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:41723
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43091 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:00:38.203526 398 log_verifier.cc:126] Checking tablet 1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:38.330605 398 log_verifier.cc:177] Verified matching terms for 795 ops in tablet 1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:38.330926 398 log_verifier.cc:126] Checking tablet 2408d49d82e34cd48ce55337221c3757
I20240503 02:00:38.447450 398 log_verifier.cc:177] Verified matching terms for 739 ops in tablet 2408d49d82e34cd48ce55337221c3757
I20240503 02:00:38.447672 398 log_verifier.cc:126] Checking tablet 34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:38.572669 398 log_verifier.cc:177] Verified matching terms for 770 ops in tablet 34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:38.572928 398 log_verifier.cc:126] Checking tablet 4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:38.694104 398 log_verifier.cc:177] Verified matching terms for 662 ops in tablet 4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:38.694326 398 log_verifier.cc:126] Checking tablet 69736194f7514beda11cce1be5711734
I20240503 02:00:38.800679 398 log_verifier.cc:177] Verified matching terms for 672 ops in tablet 69736194f7514beda11cce1be5711734
I20240503 02:00:38.800900 398 log_verifier.cc:126] Checking tablet 6fd96db9cd364050915f5333624610a2
I20240503 02:00:38.937891 398 log_verifier.cc:177] Verified matching terms for 780 ops in tablet 6fd96db9cd364050915f5333624610a2
I20240503 02:00:38.938155 398 log_verifier.cc:126] Checking tablet 701e2c3915444654989c3a817ff7b431
I20240503 02:00:39.063338 398 log_verifier.cc:177] Verified matching terms for 708 ops in tablet 701e2c3915444654989c3a817ff7b431
I20240503 02:00:39.063570 398 log_verifier.cc:126] Checking tablet 77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:39.195505 398 log_verifier.cc:177] Verified matching terms for 693 ops in tablet 77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:39.195731 398 log_verifier.cc:126] Checking tablet 8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:39.310184 398 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:39.310384 398 log_verifier.cc:126] Checking tablet 9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:39.415493 398 log_verifier.cc:177] Verified matching terms for 666 ops in tablet 9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:39.415721 398 log_verifier.cc:126] Checking tablet a39950d966ac43b890d417c3faa5992b
I20240503 02:00:39.537835 398 log_verifier.cc:177] Verified matching terms for 759 ops in tablet a39950d966ac43b890d417c3faa5992b
I20240503 02:00:39.538066 398 log_verifier.cc:126] Checking tablet a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:39.657327 398 log_verifier.cc:177] Verified matching terms for 719 ops in tablet a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:39.657536 398 log_verifier.cc:126] Checking tablet b47074508be1424f9c1686bcc6526698
I20240503 02:00:39.673458 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet b47074508be1424f9c1686bcc6526698
I20240503 02:00:39.673534 398 log_verifier.cc:126] Checking tablet c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:39.792227 398 log_verifier.cc:177] Verified matching terms for 770 ops in tablet c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:39.792451 398 log_verifier.cc:126] Checking tablet ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:39.912428 398 log_verifier.cc:177] Verified matching terms for 760 ops in tablet ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:39.912662 398 log_verifier.cc:126] Checking tablet f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:40.028668 398 log_verifier.cc:177] Verified matching terms for 689 ops in tablet f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:40.030613 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--builtin_ntp_servers=127.0.99.148:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:40.141942 11419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:40.142128 11419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:40.142174 11419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:40.145215 11419 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:40.145274 11419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:40.145361 11419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:40.149040 11419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36947
--never_fsync=true
--heap_profile_path=/tmp/kudu.11419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:40.149861 11419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:40.150974 11419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:00:40.159700 11419 server_base.cc:1034] running on GCE node
W20240503 02:00:40.160419 11425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.161212 11424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:40.161974 11427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:40.162519 11419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:40.163081 11419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:40.164242 11419 hybrid_clock.cc:648] HybridClock initialized: now 1714701640164232 us; error 25 us; skew 500 ppm
I20240503 02:00:40.166169 11419 webserver.cc:460] Webserver started at http://127.0.99.136:37425/ using document root <none> and password file <none>
I20240503 02:00:40.166679 11419 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:40.166731 11419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:40.166872 11419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:40.168505 11419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
format_stamp: "Formatted at 2024-05-03 02:00:40 on dist-test-slave-fvr5"
I20240503 02:00:40.168954 11419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
format_stamp: "Formatted at 2024-05-03 02:00:40 on dist-test-slave-fvr5"
I20240503 02:00:40.172361 11419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:40.174520 11433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.175550 11419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:40.175660 11419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
format_stamp: "Formatted at 2024-05-03 02:00:40 on dist-test-slave-fvr5"
I20240503 02:00:40.175781 11419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:40.192430 11419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:40.193033 11419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:40.193187 11419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:40.193701 11419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:40.194666 11419 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:40.194717 11419 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.194764 11419 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:40.194794 11419 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:40.203334 11419 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:37347
I20240503 02:00:40.203359 11546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:37347 every 8 connection(s)
I20240503 02:00:40.204250 11419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:40.206455 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 11419
I20240503 02:00:40.206574 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:00:40.213163 11547 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36947
I20240503 02:00:40.213366 11547 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:40.213778 11547 heartbeater.cc:511] Master 127.0.99.190:36947 requested a full tablet report, sending...
I20240503 02:00:40.214730 6783 ts_manager.cc:194] Registered new tserver with Master: 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347)
I20240503 02:00:40.215499 6783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:58767
I20240503 02:00:40.398310 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.399652 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.400641 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:40.407645 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:40.411921 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.414057 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.417186 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.441753 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.527523 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.530090 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.533243 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:40.534695 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.534937 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:40.543663 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.544777 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.558714 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.606283 7195 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 708, Committed index: 708, Last appended: 2.708, Last appended by leader: 137, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:40.607698 8290 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 2 index: 708. Preceding OpId from leader: term: 2 index: 709. (index mismatch)
I20240503 02:00:40.608142 11652 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 709, Last known committed idx: 708, Time since last communication: 0.000s
I20240503 02:00:40.609764 11416 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Committing config change with OpId 2.709: config changed from index 168 to 709, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
W20240503 02:00:40.612022 7129 consensus_peers.cc:485] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de -> Peer 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347): Couldn't send request to peer 13a76df7cf5c4f1c8c29382b60dfa2f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 701e2c3915444654989c3a817ff7b431. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:40.612112 8290 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Committing config change with OpId 2.709: config changed from index 168 to 709, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.612180 6787 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 168 to 709, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.638463 7064 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 693, Committed index: 693, Last appended: 1.693, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:40.639842 8422 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Refusing update from remote peer 69a1ad6834cc4c0dbd0436227b1f9dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240503 02:00:40.640252 11675 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240503 02:00:40.641933 10386 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Committing config change with OpId 1.694: config changed from index 303 to 694, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
W20240503 02:00:40.642830 6998 consensus_peers.cc:485] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb -> Peer 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347): Couldn't send request to peer 13a76df7cf5c4f1c8c29382b60dfa2f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77ca27caa72e415ebed755eb8a11d601. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:40.644507 8422 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 303 to 694, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.645047 6786 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb reported cstate change: config changed from index 303 to 694, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" committed_config { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.661609 8290 consensus_queue.cc:237] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:40.663105 8553 raft_consensus.cc:1261] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Refusing update from remote peer 8ac2d67357ff41b39ea5317b0874cd4c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:40.663496 11418 consensus_queue.cc:1035] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:40.665341 11418 raft_consensus.cc:2932] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
W20240503 02:00:40.666450 8224 consensus_peers.cc:485] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c -> Peer 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347): Couldn't send request to peer 13a76df7cf5c4f1c8c29382b60dfa2f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b47074508be1424f9c1686bcc6526698. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:40.667407 8553 raft_consensus.cc:2932] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.667948 6787 catalog_manager.cc:5484] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c reported cstate change: config changed from index -1 to 3, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.687851 6931 consensus_queue.cc:237] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 739, Committed index: 739, Last appended: 1.739, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:40.690516 7195 raft_consensus.cc:1261] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Refusing update from remote peer faa3c52a3f9b4bbda3ca9743ed432a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 739. Preceding OpId from leader: term: 1 index: 740. (index mismatch)
I20240503 02:00:40.690924 11716 consensus_queue.cc:1035] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 740, Last known committed idx: 739, Time since last communication: 0.000s
I20240503 02:00:40.692667 11039 raft_consensus.cc:2932] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Committing config change with OpId 1.740: config changed from index -1 to 740, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.693048 7195 raft_consensus.cc:2932] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Committing config change with OpId 1.740: config changed from index -1 to 740, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New config: { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
W20240503 02:00:40.693997 6867 consensus_peers.cc:485] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 -> Peer 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347): Couldn't send request to peer 13a76df7cf5c4f1c8c29382b60dfa2f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2408d49d82e34cd48ce55337221c3757. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:40.695530 6786 catalog_manager.cc:5484] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 reported cstate change: config changed from index -1 to 740, VOTER 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" committed_config { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:40.750034 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.756408 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.757170 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.758113 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.758183 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:40.759287 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.775633 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.776315 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:40.855026 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.862658 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.864666 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.867803 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:40.872920 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.881378 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.885624 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:40.891739 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.972597 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.975176 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.976882 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.979939 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:40.986577 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:40.987269 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:40.988441 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.004207 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:41.070214 11850 ts_tablet_manager.cc:902] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Initiating tablet copy from peer a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815)
I20240503 02:00:41.071168 11850 tablet_copy_client.cc:308] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45815
I20240503 02:00:41.079640 7214 tablet_copy_service.cc:140] P a648a1aaf5eb44faa930c687aac928de: Received BeginTabletCopySession request for tablet 701e2c3915444654989c3a817ff7b431 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 ({username='slave'} at 127.0.99.136:34507)
I20240503 02:00:41.079789 7214 tablet_copy_service.cc:161] P a648a1aaf5eb44faa930c687aac928de: Beginning new tablet copy session on tablet 701e2c3915444654989c3a817ff7b431 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 at {username='slave'} at 127.0.99.136:34507: session id = 13a76df7cf5c4f1c8c29382b60dfa2f9-701e2c3915444654989c3a817ff7b431
I20240503 02:00:41.080822 7214 tablet_copy_source_session.cc:205] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:41.084619 11850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701e2c3915444654989c3a817ff7b431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:41.089759 11850 tablet_copy_client.cc:783] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:41.089949 11850 tablet_copy_client.cc:647] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:41.095461 11850 tablet_copy_client.cc:523] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:41.098423 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.098841 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.104689 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.107437 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:41.112164 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.113551 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.126874 11850 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap starting.
I20240503 02:00:41.127405 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:41.139084 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.149420 11872 ts_tablet_manager.cc:902] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Initiating tablet copy from peer 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587)
I20240503 02:00:41.151686 11872 tablet_copy_client.cc:308] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:46587
I20240503 02:00:41.156045 7084 tablet_copy_service.cc:140] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Received BeginTabletCopySession request for tablet 77ca27caa72e415ebed755eb8a11d601 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 ({username='slave'} at 127.0.99.136:49713)
I20240503 02:00:41.156169 7084 tablet_copy_service.cc:161] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Beginning new tablet copy session on tablet 77ca27caa72e415ebed755eb8a11d601 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 at {username='slave'} at 127.0.99.136:49713: session id = 13a76df7cf5c4f1c8c29382b60dfa2f9-77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:41.157095 7084 tablet_copy_source_session.cc:205] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:41.157909 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ca27caa72e415ebed755eb8a11d601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:41.160059 11872 tablet_copy_client.cc:783] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:41.160207 11872 tablet_copy_client.cc:647] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:41.163044 11872 tablet_copy_client.cc:523] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:41.164618 11872 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap starting.
I20240503 02:00:41.186764 11883 ts_tablet_manager.cc:902] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Initiating tablet copy from peer faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253)
I20240503 02:00:41.187500 11883 tablet_copy_client.cc:308] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:45253
I20240503 02:00:41.192441 6953 tablet_copy_service.cc:140] P faa3c52a3f9b4bbda3ca9743ed432a08: Received BeginTabletCopySession request for tablet 2408d49d82e34cd48ce55337221c3757 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 ({username='slave'} at 127.0.99.136:50041)
I20240503 02:00:41.192550 6953 tablet_copy_service.cc:161] P faa3c52a3f9b4bbda3ca9743ed432a08: Beginning new tablet copy session on tablet 2408d49d82e34cd48ce55337221c3757 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 at {username='slave'} at 127.0.99.136:50041: session id = 13a76df7cf5c4f1c8c29382b60dfa2f9-2408d49d82e34cd48ce55337221c3757
I20240503 02:00:41.193521 6953 tablet_copy_source_session.cc:205] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:41.194612 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2408d49d82e34cd48ce55337221c3757. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:41.197110 11883 tablet_copy_client.cc:783] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:41.197361 11883 tablet_copy_client.cc:647] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:41.201293 11883 tablet_copy_client.cc:523] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:41.202606 11883 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap starting.
I20240503 02:00:41.206568 11850 log.cc:824] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:41.217510 11547 heartbeater.cc:503] Master 127.0.99.190:36947 was elected leader, sending a full tablet report...
I20240503 02:00:41.223412 11899 ts_tablet_manager.cc:902] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Initiating tablet copy from peer 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
I20240503 02:00:41.224292 11899 tablet_copy_client.cc:308] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:35021
I20240503 02:00:41.228961 8311 tablet_copy_service.cc:140] P 8ac2d67357ff41b39ea5317b0874cd4c: Received BeginTabletCopySession request for tablet b47074508be1424f9c1686bcc6526698 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 ({username='slave'} at 127.0.99.136:42131)
I20240503 02:00:41.229128 8311 tablet_copy_service.cc:161] P 8ac2d67357ff41b39ea5317b0874cd4c: Beginning new tablet copy session on tablet b47074508be1424f9c1686bcc6526698 from peer 13a76df7cf5c4f1c8c29382b60dfa2f9 at {username='slave'} at 127.0.99.136:42131: session id = 13a76df7cf5c4f1c8c29382b60dfa2f9-b47074508be1424f9c1686bcc6526698
I20240503 02:00:41.230540 8311 tablet_copy_source_session.cc:205] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:41.231726 11899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47074508be1424f9c1686bcc6526698. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:41.234189 11899 tablet_copy_client.cc:783] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:41.236238 11899 tablet_copy_client.cc:647] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:41.237519 11899 tablet_copy_client.cc:523] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:41.238752 11899 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap starting.
I20240503 02:00:41.248538 11899 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:41.248971 11899 tablet_bootstrap.cc:492] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap complete.
I20240503 02:00:41.249382 11899 ts_tablet_manager.cc:1367] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:00:41.252290 11899 raft_consensus.cc:348] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.257604 11899 raft_consensus.cc:729] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Initialized, Role: FOLLOWER
I20240503 02:00:41.258234 11899 consensus_queue.cc:260] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12549acb31c04923a95111409fc4bad7" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41723 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.259272 11899 ts_tablet_manager.cc:1398] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240503 02:00:41.263823 8311 tablet_copy_service.cc:342] P 8ac2d67357ff41b39ea5317b0874cd4c: Request end of tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-b47074508be1424f9c1686bcc6526698 received from {username='slave'} at 127.0.99.136:42131
I20240503 02:00:41.263932 8311 tablet_copy_service.cc:433] P 8ac2d67357ff41b39ea5317b0874cd4c: ending tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-b47074508be1424f9c1686bcc6526698 on tablet b47074508be1424f9c1686bcc6526698 with peer 13a76df7cf5c4f1c8c29382b60dfa2f9
I20240503 02:00:41.278234 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.279131 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.283330 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.303376 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.316067 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:41.321130 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.355625 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:41.369259 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.498963 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.512508 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.513254 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.514205 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.548182 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.548332 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.549796 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:41.562410 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:41.691366 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.714215 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:41.717746 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.737854 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:41.746897 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.749233 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:41.756722 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:41.764264 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:41.821224 11501 raft_consensus.cc:1203] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 02:00:41.839434 11872 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=694 overwritten=0 applied=694 ignored=0} inserts{seen=1364 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:41.839849 11872 tablet_bootstrap.cc:492] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap complete.
I20240503 02:00:41.840137 11872 ts_tablet_manager.cc:1367] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent bootstrapping tablet: real 0.676s user 0.432s sys 0.054s
I20240503 02:00:41.840786 11872 raft_consensus.cc:348] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.840921 11872 raft_consensus.cc:729] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Initialized, Role: FOLLOWER
I20240503 02:00:41.841061 11872 consensus_queue.cc:260] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.841447 11872 ts_tablet_manager.cc:1398] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:41.842128 7084 tablet_copy_service.cc:342] P 69a1ad6834cc4c0dbd0436227b1f9dbb: Request end of tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-77ca27caa72e415ebed755eb8a11d601 received from {username='slave'} at 127.0.99.136:49713
I20240503 02:00:41.842212 7084 tablet_copy_service.cc:433] P 69a1ad6834cc4c0dbd0436227b1f9dbb: ending tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-77ca27caa72e415ebed755eb8a11d601 on tablet 77ca27caa72e415ebed755eb8a11d601 with peer 13a76df7cf5c4f1c8c29382b60dfa2f9
I20240503 02:00:41.867136 11850 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=709 overwritten=0 applied=709 ignored=0} inserts{seen=1337 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:41.867453 11850 tablet_bootstrap.cc:492] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap complete.
I20240503 02:00:41.867708 11850 ts_tablet_manager.cc:1367] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent bootstrapping tablet: real 0.741s user 0.447s sys 0.047s
I20240503 02:00:41.868324 11850 raft_consensus.cc:348] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.868492 11850 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Initialized, Role: FOLLOWER
I20240503 02:00:41.868659 11850 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 709, Last appended: 2.709, Last appended by leader: 709, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.869017 11850 ts_tablet_manager.cc:1398] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:41.869606 7214 tablet_copy_service.cc:342] P a648a1aaf5eb44faa930c687aac928de: Request end of tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-701e2c3915444654989c3a817ff7b431 received from {username='slave'} at 127.0.99.136:34507
I20240503 02:00:41.869714 7214 tablet_copy_service.cc:433] P a648a1aaf5eb44faa930c687aac928de: ending tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-701e2c3915444654989c3a817ff7b431 on tablet 701e2c3915444654989c3a817ff7b431 with peer 13a76df7cf5c4f1c8c29382b60dfa2f9
I20240503 02:00:41.871205 11883 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=740 overwritten=0 applied=740 ignored=0} inserts{seen=1371 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:41.871528 11883 tablet_bootstrap.cc:492] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Bootstrap complete.
I20240503 02:00:41.871814 11883 ts_tablet_manager.cc:1367] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent bootstrapping tablet: real 0.669s user 0.477s sys 0.047s
I20240503 02:00:41.872318 11883 raft_consensus.cc:348] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.872438 11883 raft_consensus.cc:729] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Initialized, Role: FOLLOWER
I20240503 02:00:41.872552 11883 consensus_queue.cc:260] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 740, Last appended: 1.740, Last appended by leader: 740, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:41.872817 11883 ts_tablet_manager.cc:1398] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:41.873450 6953 tablet_copy_service.cc:342] P faa3c52a3f9b4bbda3ca9743ed432a08: Request end of tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-2408d49d82e34cd48ce55337221c3757 received from {username='slave'} at 127.0.99.136:50041
I20240503 02:00:41.873539 6953 tablet_copy_service.cc:433] P faa3c52a3f9b4bbda3ca9743ed432a08: ending tablet copy session 13a76df7cf5c4f1c8c29382b60dfa2f9-2408d49d82e34cd48ce55337221c3757 on tablet 2408d49d82e34cd48ce55337221c3757 with peer 13a76df7cf5c4f1c8c29382b60dfa2f9
I20240503 02:00:42.065341 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.070456 11501 raft_consensus.cc:1203] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.739->[1.740-1.740] Dedup: 1.740->[]
I20240503 02:00:42.075807 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.076411 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.080665 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.084749 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.094986 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.099313 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.114106 11501 raft_consensus.cc:1203] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.708->[2.709-2.709] Dedup: 2.709->[]
I20240503 02:00:42.115818 6913 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:42.137130 6931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757"
dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59816
I20240503 02:00:42.137269 6931 raft_consensus.cc:595] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:42.188432 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.189605 11716 raft_consensus.cc:979] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: : Instructing follower a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:42.189702 11716 raft_consensus.cc:1067] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Signalling peer a648a1aaf5eb44faa930c687aac928de to start an election
I20240503 02:00:42.190109 7195 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
from {username='slave'} at 127.0.99.129:39119
I20240503 02:00:42.190624 7195 raft_consensus.cc:482] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.190702 7195 raft_consensus.cc:3037] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.192080 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.192442 7195 raft_consensus.cc:504] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.192878 7195 leader_election.cc:288] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Requested vote from peers faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253), 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136:37347)
I20240503 02:00:42.194113 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 740 } } ignore_live_leader: true dest_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
I20240503 02:00:42.194327 11501 raft_consensus.cc:3037] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.195549 11501 raft_consensus.cc:2454] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:42.196238 7129 leader_election.cc:302] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a76df7cf5c4f1c8c29382b60dfa2f9, a648a1aaf5eb44faa930c687aac928de; no voters:
I20240503 02:00:42.196391 11416 raft_consensus.cc:2781] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:42.196538 11416 raft_consensus.cc:686] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Becoming Leader. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:42.196666 11416 consensus_queue.cc:237] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 740, Committed index: 740, Last appended: 1.740, Last appended by leader: 740, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.197515 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2408d49d82e34cd48ce55337221c3757" candidate_uuid: "a648a1aaf5eb44faa930c687aac928de" candidate_term: 2 candidate_status { last_received { term: 1 index: 740 } } ignore_live_leader: true dest_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08"
I20240503 02:00:42.197638 6931 raft_consensus.cc:3032] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:42.197678 6931 raft_consensus.cc:729] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa3c52a3f9b4bbda3ca9743ed432a08, State: Running, Role: LEADER
I20240503 02:00:42.197829 6931 consensus_queue.cc:260] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 740, Committed index: 740, Last appended: 1.740, Last appended by leader: 740, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.198004 6931 raft_consensus.cc:3037] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.199215 6931 raft_consensus.cc:2454] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a648a1aaf5eb44faa930c687aac928de in term 2.
I20240503 02:00:42.200289 11501 raft_consensus.cc:1203] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.693->[1.694-1.694] Dedup: 1.694->[]
I20240503 02:00:42.201326 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.203414 7175 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:42.203490 6787 catalog_manager.cc:5484] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de reported cstate change: term changed from 1 to 2, leader changed from faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) to a648a1aaf5eb44faa930c687aac928de (127.0.99.131). New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 740 OBSOLETE_local: false peers { permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 } } peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.211503 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.220562 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.220593 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.223070 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.316234 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.316627 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.324065 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.327160 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.328665 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.329322 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.335528 7175 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:42.354872 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.390774 8290 consensus_queue.cc:237] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.392007 11501 raft_consensus.cc:1261] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Refusing update from remote peer 8ac2d67357ff41b39ea5317b0874cd4c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:42.392423 11698 consensus_queue.cc:1035] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:42.394866 12137 raft_consensus.cc:2932] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.395360 11501 raft_consensus.cc:2932] T b47074508be1424f9c1686bcc6526698 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.397153 6787 catalog_manager.cc:5484] T b47074508be1424f9c1686bcc6526698 P 8ac2d67357ff41b39ea5317b0874cd4c reported cstate change: config changed from index 3 to 4, VOTER 12549acb31c04923a95111409fc4bad7 (127.0.99.135) evicted. New cstate: current_term: 2 leader_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.404052 8536 tablet_service.cc:1514] Processing DeleteTablet for tablet b47074508be1424f9c1686bcc6526698 with delete_type TABLET_DATA_TOMBSTONED (TS 12549acb31c04923a95111409fc4bad7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47236
I20240503 02:00:42.404543 12144 tablet_replica.cc:330] stopping tablet replica
I20240503 02:00:42.404861 12144 raft_consensus.cc:2229] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:42.405072 12144 raft_consensus.cc:2258] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:42.407042 12144 ts_tablet_manager.cc:1875] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:42.409277 12144 ts_tablet_manager.cc:1888] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:00:42.409379 12144 log.cc:1196] T b47074508be1424f9c1686bcc6526698 P 12549acb31c04923a95111409fc4bad7: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b47074508be1424f9c1686bcc6526698
I20240503 02:00:42.409965 6777 catalog_manager.cc:4830] TS 12549acb31c04923a95111409fc4bad7 (127.0.99.135:41723): tablet b47074508be1424f9c1686bcc6526698 (table TestTable [id=a0eda56e70d74d2c8d48d163c118b2e0]) successfully deleted
I20240503 02:00:42.456912 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.457613 11481 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.460426 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.461989 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.471074 7175 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:00:42.473100 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.474457 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.502727 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.525887 7195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701e2c3915444654989c3a817ff7b431"
dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60594
I20240503 02:00:42.526057 7195 raft_consensus.cc:595] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Received request to transfer leadership
I20240503 02:00:42.539624 11416 raft_consensus.cc:979] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: : Instructing follower 13a76df7cf5c4f1c8c29382b60dfa2f9 to start an election
I20240503 02:00:42.539780 11416 raft_consensus.cc:1067] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Signalling peer 13a76df7cf5c4f1c8c29382b60dfa2f9 to start an election
I20240503 02:00:42.540298 11501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "701e2c3915444654989c3a817ff7b431"
dest_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
from {username='slave'} at 127.0.99.131:48641
I20240503 02:00:42.540499 11501 raft_consensus.cc:482] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.540565 11501 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:42.541597 11501 raft_consensus.cc:504] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.542744 11501 leader_election.cc:288] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [CANDIDATE]: Term 3 election: Requested vote from peers a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815), 8ac2d67357ff41b39ea5317b0874cd4c (127.0.99.133:35021)
I20240503 02:00:42.543095 7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 709 } } ignore_live_leader: true dest_uuid: "a648a1aaf5eb44faa930c687aac928de"
I20240503 02:00:42.543100 8290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701e2c3915444654989c3a817ff7b431" candidate_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 709 } } ignore_live_leader: true dest_uuid: "8ac2d67357ff41b39ea5317b0874cd4c"
I20240503 02:00:42.543217 7195 raft_consensus.cc:3032] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:00:42.543236 8290 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:42.543256 7195 raft_consensus.cc:729] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a648a1aaf5eb44faa930c687aac928de, State: Running, Role: LEADER
I20240503 02:00:42.543395 7195 consensus_queue.cc:260] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 709, Committed index: 709, Last appended: 2.709, Last appended by leader: 709, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.543536 7195 raft_consensus.cc:3037] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:00:42.544234 8290 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a76df7cf5c4f1c8c29382b60dfa2f9 in term 3.
I20240503 02:00:42.544560 7195 raft_consensus.cc:2454] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a76df7cf5c4f1c8c29382b60dfa2f9 in term 3.
I20240503 02:00:42.544605 11434 leader_election.cc:302] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a76df7cf5c4f1c8c29382b60dfa2f9, 8ac2d67357ff41b39ea5317b0874cd4c; no voters:
I20240503 02:00:42.544876 11998 raft_consensus.cc:2781] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:00:42.545042 11998 raft_consensus.cc:686] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 3 LEADER]: Becoming Leader. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Running, Role: LEADER
I20240503 02:00:42.545337 11998 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 709, Committed index: 709, Last appended: 2.709, Last appended by leader: 709, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.547569 6785 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 reported cstate change: term changed from 2 to 3, leader changed from a648a1aaf5eb44faa930c687aac928de (127.0.99.131) to 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136). New cstate: current_term: 3 leader_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" committed_config { opid_index: 709 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.580812 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.581614 11481 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.588815 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.589036 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.590939 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.595909 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.610474 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.611392 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.612565 6931 raft_consensus.cc:1261] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 740. Preceding OpId from leader: term: 2 index: 741. (index mismatch)
I20240503 02:00:42.613009 11416 consensus_queue.cc:1035] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa3c52a3f9b4bbda3ca9743ed432a08" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 45253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 741, Last known committed idx: 740, Time since last communication: 0.000s
I20240503 02:00:42.636600 11501 raft_consensus.cc:1261] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 1 index: 740. Preceding OpId from leader: term: 2 index: 741. (index mismatch)
I20240503 02:00:42.637115 12076 consensus_queue.cc:1035] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 741, Last known committed idx: 740, Time since last communication: 0.000s
I20240503 02:00:42.718760 7044 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.719208 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.720921 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.722069 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.730465 11481 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.732093 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.743799 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.744448 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.774478 7064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601"
dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51140
I20240503 02:00:42.774626 7064 raft_consensus.cc:595] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:42.789536 12272 raft_consensus.cc:979] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: : Instructing follower 13a76df7cf5c4f1c8c29382b60dfa2f9 to start an election
I20240503 02:00:42.789667 12272 raft_consensus.cc:1067] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Signalling peer 13a76df7cf5c4f1c8c29382b60dfa2f9 to start an election
I20240503 02:00:42.790086 11501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601"
dest_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9"
from {username='slave'} at 127.0.99.130:52237
I20240503 02:00:42.790218 11501 raft_consensus.cc:482] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:42.790282 11501 raft_consensus.cc:3037] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.791076 11501 raft_consensus.cc:504] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.791515 11501 leader_election.cc:288] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [CANDIDATE]: Term 2 election: Requested vote from peers 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587), 591f812606bf438fbb9bd0a977c678e6 (127.0.99.134:45311)
I20240503 02:00:42.792071 7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601" candidate_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: true dest_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb"
I20240503 02:00:42.792196 7064 raft_consensus.cc:3032] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:42.792232 7064 raft_consensus.cc:729] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69a1ad6834cc4c0dbd0436227b1f9dbb, State: Running, Role: LEADER
I20240503 02:00:42.792361 7064 consensus_queue.cc:260] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 694, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.792485 7064 raft_consensus.cc:3037] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.793445 7064 raft_consensus.cc:2454] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a76df7cf5c4f1c8c29382b60dfa2f9 in term 2.
I20240503 02:00:42.793751 11437 leader_election.cc:302] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a76df7cf5c4f1c8c29382b60dfa2f9, 69a1ad6834cc4c0dbd0436227b1f9dbb; no voters:
I20240503 02:00:42.793933 12189 raft_consensus.cc:2781] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:42.794026 12189 raft_consensus.cc:686] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 LEADER]: Becoming Leader. State: Replica: 13a76df7cf5c4f1c8c29382b60dfa2f9, State: Running, Role: LEADER
I20240503 02:00:42.794163 12189 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 694, Committed index: 694, Last appended: 1.694, Last appended by leader: 694, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:42.796981 6783 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 reported cstate change: term changed from 1 to 2, leader changed from 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) to 13a76df7cf5c4f1c8c29382b60dfa2f9 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" committed_config { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 } } peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:42.797441 8422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ca27caa72e415ebed755eb8a11d601" candidate_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: true dest_uuid: "591f812606bf438fbb9bd0a977c678e6"
I20240503 02:00:42.797569 8422 raft_consensus.cc:3037] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:42.798605 8422 raft_consensus.cc:2454] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a76df7cf5c4f1c8c29382b60dfa2f9 in term 2.
I20240503 02:00:42.829192 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.829615 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.831295 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:42.833783 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:42.833884 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:42.841316 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:42.842710 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.858047 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:42.926470 8290 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 3 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 709. Preceding OpId from leader: term: 3 index: 710. (index mismatch)
I20240503 02:00:42.927039 11998 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 710, Last known committed idx: 709, Time since last communication: 0.000s
I20240503 02:00:42.929941 7195 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 3 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 709. Preceding OpId from leader: term: 3 index: 710. (index mismatch)
I20240503 02:00:42.930403 11998 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 710, Last known committed idx: 709, Time since last communication: 0.000s
I20240503 02:00:43.142050 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.142938 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.147965 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.152873 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.153630 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.157528 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.164947 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.169009 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.202946 7064 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 2 index: 695. (index mismatch)
I20240503 02:00:43.203485 11998 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a1ad6834cc4c0dbd0436227b1f9dbb" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 46587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240503 02:00:43.206895 8422 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 2 index: 695. (index mismatch)
I20240503 02:00:43.208024 12349 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240503 02:00:43.223479 7195 consensus_queue.cc:237] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 741, Committed index: 741, Last appended: 2.741, Last appended by leader: 740, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 742 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:43.224457 11501 raft_consensus.cc:1261] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Refusing update from remote peer a648a1aaf5eb44faa930c687aac928de: Log matching property violated. Preceding OpId in replica: term: 2 index: 741. Preceding OpId from leader: term: 2 index: 742. (index mismatch)
I20240503 02:00:43.224829 12076 consensus_queue.cc:1035] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 742, Last known committed idx: 741, Time since last communication: 0.000s
I20240503 02:00:43.226347 12363 raft_consensus.cc:2932] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de [term 2 LEADER]: Committing config change with OpId 2.742: config changed from index 740 to 742, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 742 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.226675 11501 raft_consensus.cc:2932] T 2408d49d82e34cd48ce55337221c3757 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 FOLLOWER]: Committing config change with OpId 2.742: config changed from index 740 to 742, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New config: { opid_index: 742 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.229207 6790 catalog_manager.cc:5484] T 2408d49d82e34cd48ce55337221c3757 P a648a1aaf5eb44faa930c687aac928de reported cstate change: config changed from index 740 to 742, VOTER faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "a648a1aaf5eb44faa930c687aac928de" committed_config { opid_index: 742 OBSOLETE_local: false peers { permanent_uuid: "a648a1aaf5eb44faa930c687aac928de" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45815 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.231935 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 2408d49d82e34cd48ce55337221c3757 with delete_type TABLET_DATA_TOMBSTONED (TS faa3c52a3f9b4bbda3ca9743ed432a08 not found in new config with opid_index 742) from {username='slave'} at 127.0.0.1:41376
I20240503 02:00:43.232297 12369 tablet_replica.cc:330] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: stopping tablet replica
I20240503 02:00:43.232427 12369 raft_consensus.cc:2229] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:43.232558 12369 raft_consensus.cc:2258] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:43.233842 12369 ts_tablet_manager.cc:1875] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:43.236027 12369 ts_tablet_manager.cc:1888] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.741
I20240503 02:00:43.236093 12369 log.cc:1196] T 2408d49d82e34cd48ce55337221c3757 P faa3c52a3f9b4bbda3ca9743ed432a08: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2408d49d82e34cd48ce55337221c3757
I20240503 02:00:43.236670 6775 catalog_manager.cc:4830] TS faa3c52a3f9b4bbda3ca9743ed432a08 (127.0.99.129:45253): tablet 2408d49d82e34cd48ce55337221c3757 (table rebalance_test_table_1 [id=923688f0034044adbaeb3caae37e4e28]) successfully deleted
I20240503 02:00:43.291689 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.296686 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.301164 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.303128 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.322535 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.323484 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.332389 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.336917 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.375572 11501 consensus_queue.cc:237] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 710, Committed index: 710, Last appended: 3.710, Last appended by leader: 709, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 711 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:43.376874 8290 raft_consensus.cc:1261] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 3 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 3 index: 710. Preceding OpId from leader: term: 3 index: 711. (index mismatch)
I20240503 02:00:43.377238 12349 consensus_queue.cc:1035] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 711, Last known committed idx: 710, Time since last communication: 0.000s
I20240503 02:00:43.378698 11998 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 3 LEADER]: Committing config change with OpId 3.711: config changed from index 709 to 711, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 711 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.379132 8290 raft_consensus.cc:2932] T 701e2c3915444654989c3a817ff7b431 P 8ac2d67357ff41b39ea5317b0874cd4c [term 3 FOLLOWER]: Committing config change with OpId 3.711: config changed from index 709 to 711, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New config: { opid_index: 711 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.381668 6790 catalog_manager.cc:5484] T 701e2c3915444654989c3a817ff7b431 P 13a76df7cf5c4f1c8c29382b60dfa2f9 reported cstate change: config changed from index 709 to 711, VOTER a648a1aaf5eb44faa930c687aac928de (127.0.99.131) evicted. New cstate: current_term: 3 leader_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" committed_config { opid_index: 711 OBSOLETE_local: false peers { permanent_uuid: "8ac2d67357ff41b39ea5317b0874cd4c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 35021 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.384296 7175 tablet_service.cc:1514] Processing DeleteTablet for tablet 701e2c3915444654989c3a817ff7b431 with delete_type TABLET_DATA_TOMBSTONED (TS a648a1aaf5eb44faa930c687aac928de not found in new config with opid_index 711) from {username='slave'} at 127.0.0.1:59270
I20240503 02:00:43.384676 12423 tablet_replica.cc:330] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: stopping tablet replica
I20240503 02:00:43.384811 12423 raft_consensus.cc:2229] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:43.384927 12423 raft_consensus.cc:2258] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:43.386534 12423 ts_tablet_manager.cc:1875] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:43.388932 12423 ts_tablet_manager.cc:1888] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.710
I20240503 02:00:43.389014 12423 log.cc:1196] T 701e2c3915444654989c3a817ff7b431 P a648a1aaf5eb44faa930c687aac928de: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/701e2c3915444654989c3a817ff7b431
I20240503 02:00:43.389611 6777 catalog_manager.cc:4830] TS a648a1aaf5eb44faa930c687aac928de (127.0.99.131:45815): tablet 701e2c3915444654989c3a817ff7b431 (table rebalance_test_table_3 [id=8a7d551be09747bf82c81fbac3feea99]) successfully deleted
I20240503 02:00:43.440385 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.441241 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.442826 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.444540 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.455168 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.460530 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.471395 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.471951 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.517822 11501 consensus_queue.cc:237] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 695, Committed index: 695, Last appended: 2.695, Last appended by leader: 694, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } }
I20240503 02:00:43.518877 8422 raft_consensus.cc:1261] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Refusing update from remote peer 13a76df7cf5c4f1c8c29382b60dfa2f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 695. Preceding OpId from leader: term: 2 index: 696. (index mismatch)
I20240503 02:00:43.519280 11998 consensus_queue.cc:1035] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 696, Last known committed idx: 695, Time since last communication: 0.000s
I20240503 02:00:43.520710 12349 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 [term 2 LEADER]: Committing config change with OpId 2.696: config changed from index 694 to 696, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.521109 8422 raft_consensus.cc:2932] T 77ca27caa72e415ebed755eb8a11d601 P 591f812606bf438fbb9bd0a977c678e6 [term 2 FOLLOWER]: Committing config change with OpId 2.696: config changed from index 694 to 696, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New config: { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.523036 6787 catalog_manager.cc:5484] T 77ca27caa72e415ebed755eb8a11d601 P 13a76df7cf5c4f1c8c29382b60dfa2f9 reported cstate change: config changed from index 694 to 696, VOTER 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" committed_config { opid_index: 696 OBSOLETE_local: false peers { permanent_uuid: "591f812606bf438fbb9bd0a977c678e6" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 45311 } } peers { permanent_uuid: "13a76df7cf5c4f1c8c29382b60dfa2f9" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 37347 } } }
I20240503 02:00:43.525553 7044 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ca27caa72e415ebed755eb8a11d601 with delete_type TABLET_DATA_TOMBSTONED (TS 69a1ad6834cc4c0dbd0436227b1f9dbb not found in new config with opid_index 696) from {username='slave'} at 127.0.0.1:52842
I20240503 02:00:43.525887 12477 tablet_replica.cc:330] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: stopping tablet replica
I20240503 02:00:43.526095 12477 raft_consensus.cc:2229] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:43.526216 12477 raft_consensus.cc:2258] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:43.527861 12477 ts_tablet_manager.cc:1875] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:43.530175 12477 ts_tablet_manager.cc:1888] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.695
I20240503 02:00:43.530241 12477 log.cc:1196] T 77ca27caa72e415ebed755eb8a11d601 P 69a1ad6834cc4c0dbd0436227b1f9dbb: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:43.530928 6778 catalog_manager.cc:4830] TS 69a1ad6834cc4c0dbd0436227b1f9dbb (127.0.99.130:46587): tablet 77ca27caa72e415ebed755eb8a11d601 (table rebalance_test_table_2 [id=fb3785aaf951425289547f6daa48f30c]) successfully deleted
I20240503 02:00:43.598028 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.598244 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.607329 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.608907 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.611303 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.612871 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.617919 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.629997 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.724015 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.726419 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.726694 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.727391 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.732791 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.733124 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.743362 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.764959 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.847174 8139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.848707 8536 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:43.851143 8271 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:00:43.858098 7044 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:00:43.859376 7175 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.867895 11481 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:00:43.880878 6913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:00:43.889665 8404 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
cc32e3225a044351a09cc29d3e9fff23 | 127.0.99.190:36947 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43091 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
07bf298e40944e249d6ed0d50087aff7 | 127.0.99.132:41379 | HEALTHY | <none> | 0 | 0
12549acb31c04923a95111409fc4bad7 | 127.0.99.135:41723 | HEALTHY | <none> | 0 | 0
13a76df7cf5c4f1c8c29382b60dfa2f9 | 127.0.99.136:37347 | HEALTHY | <none> | 2 | 0
591f812606bf438fbb9bd0a977c678e6 | 127.0.99.134:45311 | HEALTHY | <none> | 2 | 0
69a1ad6834cc4c0dbd0436227b1f9dbb | 127.0.99.130:46587 | HEALTHY | <none> | 3 | 0
8ac2d67357ff41b39ea5317b0874cd4c | 127.0.99.133:35021 | HEALTHY | <none> | 1 | 0
a648a1aaf5eb44faa930c687aac928de | 127.0.99.131:45815 | HEALTHY | <none> | 4 | 0
faa3c52a3f9b4bbda3ca9743ed432a08 | 127.0.99.129:45253 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:45253
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:46587
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:45815
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41379
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:35021
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:45311
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:41723
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:37347
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:45253
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:46587
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:45815
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41379
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:35021
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:45311
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:41723
server_dump_info_path | /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:37347
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43091 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:00:43.911587 398 log_verifier.cc:126] Checking tablet 1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:44.039094 398 log_verifier.cc:177] Verified matching terms for 795 ops in tablet 1d9d6747dd9146f99ec8e06ef37b0c87
I20240503 02:00:44.039307 398 log_verifier.cc:126] Checking tablet 2408d49d82e34cd48ce55337221c3757
I20240503 02:00:44.155532 398 log_verifier.cc:177] Verified matching terms for 742 ops in tablet 2408d49d82e34cd48ce55337221c3757
I20240503 02:00:44.155786 398 log_verifier.cc:126] Checking tablet 34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:44.274574 398 log_verifier.cc:177] Verified matching terms for 770 ops in tablet 34c7315583b84d67853e2d5de4d390d5
I20240503 02:00:44.274808 398 log_verifier.cc:126] Checking tablet 4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:44.383190 398 log_verifier.cc:177] Verified matching terms for 662 ops in tablet 4ad4302f3e0d4e1191aff07a2f39765a
I20240503 02:00:44.383378 398 log_verifier.cc:126] Checking tablet 69736194f7514beda11cce1be5711734
I20240503 02:00:44.485016 398 log_verifier.cc:177] Verified matching terms for 672 ops in tablet 69736194f7514beda11cce1be5711734
I20240503 02:00:44.485211 398 log_verifier.cc:126] Checking tablet 6fd96db9cd364050915f5333624610a2
I20240503 02:00:44.604549 398 log_verifier.cc:177] Verified matching terms for 780 ops in tablet 6fd96db9cd364050915f5333624610a2
I20240503 02:00:44.604789 398 log_verifier.cc:126] Checking tablet 701e2c3915444654989c3a817ff7b431
I20240503 02:00:44.717526 398 log_verifier.cc:177] Verified matching terms for 711 ops in tablet 701e2c3915444654989c3a817ff7b431
I20240503 02:00:44.717787 398 log_verifier.cc:126] Checking tablet 77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:44.824927 398 log_verifier.cc:177] Verified matching terms for 696 ops in tablet 77ca27caa72e415ebed755eb8a11d601
I20240503 02:00:44.825136 398 log_verifier.cc:126] Checking tablet 8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:44.927042 398 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 8af5c8713282498abc8cf3606a7d2279
I20240503 02:00:44.927248 398 log_verifier.cc:126] Checking tablet 9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:45.029652 398 log_verifier.cc:177] Verified matching terms for 666 ops in tablet 9c0637b4a4864fd4807a118ab4c45af3
I20240503 02:00:45.029879 398 log_verifier.cc:126] Checking tablet a39950d966ac43b890d417c3faa5992b
I20240503 02:00:45.147166 398 log_verifier.cc:177] Verified matching terms for 759 ops in tablet a39950d966ac43b890d417c3faa5992b
I20240503 02:00:45.147405 398 log_verifier.cc:126] Checking tablet a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:45.255985 398 log_verifier.cc:177] Verified matching terms for 719 ops in tablet a3abce62955548a9b7e745108b60dd1b
I20240503 02:00:45.256224 398 log_verifier.cc:126] Checking tablet b47074508be1424f9c1686bcc6526698
I20240503 02:00:45.268369 398 log_verifier.cc:177] Verified matching terms for 4 ops in tablet b47074508be1424f9c1686bcc6526698
I20240503 02:00:45.268436 398 log_verifier.cc:126] Checking tablet c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:45.386286 398 log_verifier.cc:177] Verified matching terms for 770 ops in tablet c77c0d649d2c4ae39e0ade14fbe06c9e
I20240503 02:00:45.386545 398 log_verifier.cc:126] Checking tablet ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:45.499513 398 log_verifier.cc:177] Verified matching terms for 760 ops in tablet ef695f025d7c479b98b9e8b6ee4fc4e3
I20240503 02:00:45.499822 398 log_verifier.cc:126] Checking tablet f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:45.605204 398 log_verifier.cc:177] Verified matching terms for 689 ops in tablet f40bc52bd2064bcfaa79ab4492de1320
I20240503 02:00:45.606909 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6851
I20240503 02:00:45.615396 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6982
I20240503 02:00:45.622431 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 7113
I20240503 02:00:45.629280 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 8077
I20240503 02:00:45.635133 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 8208
I20240503 02:00:45.641410 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 8341
I20240503 02:00:45.647226 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 8473
I20240503 02:00:45.653035 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 11419
I20240503 02:00:45.658687 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 6760
2024-05-03T02:00:45Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (40844 ms)
[----------] 1 test from RebalanceParamTest (40844 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 02:00:45.717500 398 test_util.cc:268] Using random seed: 377298934
I20240503 02:00:45.718298 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:00:45.718335 398 ts_itest-base.cc:113] --------------
I20240503 02:00:45.718354 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:00:45.718372 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:00:45.718400 398 ts_itest-base.cc:116] --------------
2024-05-03T02:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:00:45Z Disabled control of system clock
I20240503 02:00:45.726490 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37619
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37619
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:00:45.833400 12603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:45.833597 12603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:45.833634 12603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:45.836534 12603 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:45.836588 12603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:00:45.836618 12603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:45.836642 12603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:00:45.836665 12603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:00:45.840142 12603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37619
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:45.840934 12603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:45.841951 12603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:45.849519 12608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:45.850302 12611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:45.851061 12609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:45.851959 12603 server_base.cc:1034] running on GCE node
I20240503 02:00:45.852386 12603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:45.853231 12603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:45.854386 12603 hybrid_clock.cc:648] HybridClock initialized: now 1714701645854367 us; error 43 us; skew 500 ppm
I20240503 02:00:45.856202 12603 webserver.cc:460] Webserver started at http://127.0.99.190:45733/ using document root <none> and password file <none>
I20240503 02:00:45.856727 12603 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:45.856786 12603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:45.856977 12603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:45.858565 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e267e0608b874bc396051ef2febea58d"
format_stamp: "Formatted at 2024-05-03 02:00:45 on dist-test-slave-fvr5"
I20240503 02:00:45.859010 12603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e267e0608b874bc396051ef2febea58d"
format_stamp: "Formatted at 2024-05-03 02:00:45 on dist-test-slave-fvr5"
I20240503 02:00:45.862135 12603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240503 02:00:45.864149 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:45.865135 12603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:45.865245 12603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e267e0608b874bc396051ef2febea58d"
format_stamp: "Formatted at 2024-05-03 02:00:45 on dist-test-slave-fvr5"
I20240503 02:00:45.865324 12603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:45.893736 12603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:45.894351 12603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:45.894521 12603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:45.900943 12603 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37619
I20240503 02:00:45.900952 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37619 every 8 connection(s)
I20240503 02:00:45.901890 12603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:00:45.904335 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:45.909034 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d: Bootstrap starting.
I20240503 02:00:45.911065 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:45.911454 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 12603
I20240503 02:00:45.911576 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:00:45.911713 12670 log.cc:824] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:45.913600 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d: No bootstrap required, opened a new log
I20240503 02:00:45.916074 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } }
I20240503 02:00:45.916241 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:45.916301 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e267e0608b874bc396051ef2febea58d, State: Initialized, Role: FOLLOWER
I20240503 02:00:45.916936 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } }
I20240503 02:00:45.917156 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:00:45.917224 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:00:45.917326 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:45.918303 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } }
I20240503 02:00:45.918637 12670 leader_election.cc:302] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e267e0608b874bc396051ef2febea58d; no voters:
I20240503 02:00:45.918921 12670 leader_election.cc:288] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:00:45.919018 12675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:45.919183 12675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [term 1 LEADER]: Becoming Leader. State: Replica: e267e0608b874bc396051ef2febea58d, State: Running, Role: LEADER
I20240503 02:00:45.919510 12675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } }
I20240503 02:00:45.920001 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:00:45.920655 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e267e0608b874bc396051ef2febea58d. Latest consensus state: current_term: 1 leader_uuid: "e267e0608b874bc396051ef2febea58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } } }
I20240503 02:00:45.920866 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:45.921209 12679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:00:45.921204 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e267e0608b874bc396051ef2febea58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e267e0608b874bc396051ef2febea58d" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37619 } } }
I20240503 02:00:45.921434 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d [sys.catalog]: This master's current role is: LEADER
I20240503 02:00:45.924204 12679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:00:45.928689 12679 catalog_manager.cc:1347] Generated new cluster ID: b5231c1bcebd497b84ea7c2ac08ade7e
I20240503 02:00:45.928754 12679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:00:45.946727 12679 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:00:45.947988 12679 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:00:45.960886 12679 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e267e0608b874bc396051ef2febea58d: Generated new TSK 0
I20240503 02:00:45.961555 12679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:00:45.975193 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:46.078807 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:46.079018 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:46.079062 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:46.082157 12694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:46.082226 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:46.082302 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:00:46.086410 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:46.087276 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:46.088369 12694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:46.096763 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.097528 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.098315 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:46.099290 12694 server_base.cc:1034] running on GCE node
I20240503 02:00:46.099659 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:46.100232 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:46.101374 12694 hybrid_clock.cc:648] HybridClock initialized: now 1714701646101362 us; error 26 us; skew 500 ppm
I20240503 02:00:46.103178 12694 webserver.cc:460] Webserver started at http://127.0.99.129:36879/ using document root <none> and password file <none>
I20240503 02:00:46.103691 12694 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:46.103744 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:46.103964 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:46.105515 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bbba999bf95446a68b6586608b33ee16"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.105983 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bbba999bf95446a68b6586608b33ee16"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.109149 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 02:00:46.111256 12708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.112236 12694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:46.112344 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bbba999bf95446a68b6586608b33ee16"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.112423 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:46.142776 12694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:46.143383 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:46.143543 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:46.144228 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:46.145164 12694 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:46.145220 12694 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.145287 12694 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:46.145318 12694 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.153949 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33445
I20240503 02:00:46.153970 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33445 every 8 connection(s)
I20240503 02:00:46.154835 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:00:46.160653 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 12694
I20240503 02:00:46.160777 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:00:46.163029 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:46.163488 12822 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:46.163694 12822 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:46.164137 12822 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:46.165546 12634 ts_manager.cc:194] Registered new tserver with Master: bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:46.166667 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:46595
W20240503 02:00:46.266964 12825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:46.267184 12825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:46.267258 12825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:46.270324 12825 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:46.270396 12825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:46.270486 12825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:00:46.273960 12825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.12825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:46.274770 12825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:46.275843 12825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:46.284247 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.285012 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.285779 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:46.286701 12825 server_base.cc:1034] running on GCE node
I20240503 02:00:46.287039 12825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:46.287508 12825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:46.288678 12825 hybrid_clock.cc:648] HybridClock initialized: now 1714701646288663 us; error 28 us; skew 500 ppm
I20240503 02:00:46.295948 12825 webserver.cc:460] Webserver started at http://127.0.99.130:45231/ using document root <none> and password file <none>
I20240503 02:00:46.296473 12825 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:46.296526 12825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:46.296761 12825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:46.298638 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.299140 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.302340 12825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240503 02:00:46.304572 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.305569 12825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:00:46.305694 12825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.305783 12825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:46.317021 12825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:46.317567 12825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:46.317732 12825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:46.318231 12825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:46.319142 12825 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:46.319202 12825 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.319255 12825 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:46.319305 12825 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.327406 12825 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:43951
I20240503 02:00:46.327435 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:43951 every 8 connection(s)
I20240503 02:00:46.328289 12825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:00:46.328575 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 12825
I20240503 02:00:46.328691 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:00:46.330950 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:46.336786 12953 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:46.337011 12953 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:46.337436 12953 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:46.338323 12634 ts_manager.cc:194] Registered new tserver with Master: 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:46.338838 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:33603
W20240503 02:00:46.435603 12956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:46.435843 12956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:46.435914 12956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:46.438846 12956 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:46.438902 12956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:46.439004 12956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:00:46.442399 12956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.12956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:46.443162 12956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:46.444216 12956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:46.452533 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.453397 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.454152 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:46.455016 12956 server_base.cc:1034] running on GCE node
I20240503 02:00:46.455384 12956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:46.455909 12956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:46.457055 12956 hybrid_clock.cc:648] HybridClock initialized: now 1714701646457042 us; error 30 us; skew 500 ppm
I20240503 02:00:46.458837 12956 webserver.cc:460] Webserver started at http://127.0.99.131:46777/ using document root <none> and password file <none>
I20240503 02:00:46.459335 12956 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:46.459391 12956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:46.459563 12956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:46.461236 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "22b582a5f4774861a5619c51198d39c5"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.461694 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "22b582a5f4774861a5619c51198d39c5"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.464870 12956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:46.466823 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.467718 12956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:46.467844 12956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "22b582a5f4774861a5619c51198d39c5"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.467926 12956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:46.517131 12956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:46.517732 12956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:46.517881 12956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:46.518429 12956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:46.519299 12956 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:46.519344 12956 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.519390 12956 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:46.519423 12956 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.528144 12956 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43679
I20240503 02:00:46.528187 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43679 every 8 connection(s)
I20240503 02:00:46.529032 12956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:00:46.536283 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 12956
I20240503 02:00:46.536392 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:00:46.537181 13084 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:46.537398 13084 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:46.537930 13084 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:46.538455 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:46.538888 12634 ts_manager.cc:194] Registered new tserver with Master: 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:46.539433 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:43135
W20240503 02:00:46.643610 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:46.643831 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:46.643872 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:46.646760 13087 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:46.646809 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:46.646871 13087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:00:46.650482 13087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.13087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:46.651242 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:46.652259 13087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:46.660984 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.661772 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.662598 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:46.663393 13087 server_base.cc:1034] running on GCE node
I20240503 02:00:46.663735 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:46.664310 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:46.665454 13087 hybrid_clock.cc:648] HybridClock initialized: now 1714701646665449 us; error 26 us; skew 500 ppm
I20240503 02:00:46.667241 13087 webserver.cc:460] Webserver started at http://127.0.99.132:43775/ using document root <none> and password file <none>
I20240503 02:00:46.667747 13087 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:46.667843 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:46.668049 13087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:46.669699 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "03ced8676740485e86ebe292f8ee4175"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.670140 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "03ced8676740485e86ebe292f8ee4175"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.673472 13087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:46.675488 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.676478 13087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:46.676570 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "03ced8676740485e86ebe292f8ee4175"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.676658 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:46.712101 13087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:46.712718 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:46.712875 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:46.713402 13087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:46.714269 13087 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:46.714315 13087 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.714359 13087 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:46.714390 13087 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.722865 13087 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36157
I20240503 02:00:46.722920 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36157 every 8 connection(s)
I20240503 02:00:46.723778 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:00:46.724459 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 13087
I20240503 02:00:46.724568 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:00:46.726608 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:46.732820 13215 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:46.733026 13215 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:46.733428 13215 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:46.734270 12634 ts_manager.cc:194] Registered new tserver with Master: 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:46.734800 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39621
W20240503 02:00:46.831604 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:46.831830 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:46.831872 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:46.834813 13218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:46.834872 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:46.834939 13218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:46.838366 13218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:46.839200 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:46.840309 13218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:46.848695 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.849503 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:46.850276 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:46.851132 13218 server_base.cc:1034] running on GCE node
I20240503 02:00:46.851493 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:46.852036 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:46.853179 13218 hybrid_clock.cc:648] HybridClock initialized: now 1714701646853170 us; error 22 us; skew 500 ppm
I20240503 02:00:46.854961 13218 webserver.cc:460] Webserver started at http://127.0.99.133:46583/ using document root <none> and password file <none>
I20240503 02:00:46.855466 13218 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:46.855518 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:46.855692 13218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:46.857303 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9e39ab9b64544988b968015fe9fa06e6"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.857764 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9e39ab9b64544988b968015fe9fa06e6"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.860875 13218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:46.862867 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.863826 13218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:46.863919 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9e39ab9b64544988b968015fe9fa06e6"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:46.864143 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:46.880728 13218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:46.881248 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:46.881402 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:46.881863 13218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:46.882652 13218 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:46.882696 13218 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.882751 13218 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:46.882778 13218 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:46.891162 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37055
I20240503 02:00:46.891204 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37055 every 8 connection(s)
I20240503 02:00:46.892024 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:46.900254 13346 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:46.900449 13346 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:46.900887 13346 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:46.901718 12634 ts_manager.cc:194] Registered new tserver with Master: 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:00:46.902133 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 13218
I20240503 02:00:46.902261 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:00:46.902376 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35595
I20240503 02:00:46.904351 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:47.009289 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:47.009492 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:47.009548 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:47.012526 13349 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:47.012581 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:47.012686 13349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:47.016224 13349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.13349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:47.017019 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:47.018077 13349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.026487 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.027247 13354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.028021 13355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.028849 13349 server_base.cc:1034] running on GCE node
I20240503 02:00:47.029181 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:47.029690 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:47.030848 13349 hybrid_clock.cc:648] HybridClock initialized: now 1714701647030827 us; error 39 us; skew 500 ppm
I20240503 02:00:47.032673 13349 webserver.cc:460] Webserver started at http://127.0.99.134:37021/ using document root <none> and password file <none>
I20240503 02:00:47.033195 13349 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.033254 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.033427 13349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.035017 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.035485 13349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.038606 13349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:00:47.040701 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.041659 13349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:47.041767 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.041844 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.056711 13349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.057269 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:47.057420 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.057888 13349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.058698 13349 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.058753 13349 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.058811 13349 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.058841 13349 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.066547 13349 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40697
I20240503 02:00:47.066573 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40697 every 8 connection(s)
I20240503 02:00:47.067443 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:47.069392 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 13349
I20240503 02:00:47.069494 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:00:47.071775 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:47.076444 13477 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:47.076658 13477 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.077059 13477 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:47.078003 12634 ts_manager.cc:194] Registered new tserver with Master: d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:00:47.078804 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:44645
I20240503 02:00:47.168846 12822 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
W20240503 02:00:47.178069 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:47.178287 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:47.178341 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:47.181520 13480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:47.181587 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:47.181699 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:47.185180 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.13480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:47.186043 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:47.187166 13480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.196023 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.196596 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.197427 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.198261 13480 server_base.cc:1034] running on GCE node
I20240503 02:00:47.198655 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:47.199211 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:47.200347 13480 hybrid_clock.cc:648] HybridClock initialized: now 1714701647200326 us; error 36 us; skew 500 ppm
I20240503 02:00:47.202172 13480 webserver.cc:460] Webserver started at http://127.0.99.135:44591/ using document root <none> and password file <none>
I20240503 02:00:47.202694 13480 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.202749 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.202927 13480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.204589 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "754fc04ea9914073a56c0cf164f63240"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.205029 13480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "754fc04ea9914073a56c0cf164f63240"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.208505 13480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:00:47.210685 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.211624 13480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 02:00:47.211720 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "754fc04ea9914073a56c0cf164f63240"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.211829 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.224798 13480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.225332 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:47.225481 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.225980 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.226830 13480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.226876 13480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.226925 13480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.226955 13480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.234802 13480 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37171
I20240503 02:00:47.234845 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37171 every 8 connection(s)
I20240503 02:00:47.235714 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:47.236985 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 13480
I20240503 02:00:47.237102 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:00:47.239238 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:47.245215 13608 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:47.245445 13608 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.245967 13608 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:47.246886 12634 ts_manager.cc:194] Registered new tserver with Master: 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:00:47.247390 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:39017
I20240503 02:00:47.340634 12953 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
W20240503 02:00:47.346297 13611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:47.346498 13611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:47.346565 13611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:47.349627 13611 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:47.349696 13611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:47.349785 13611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:47.353268 13611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.13611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:47.354090 13611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:47.355144 13611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:47.363627 13616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.364391 13617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:47.365268 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:47.366115 13611 server_base.cc:1034] running on GCE node
I20240503 02:00:47.366473 13611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:47.367024 13611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:47.368175 13611 hybrid_clock.cc:648] HybridClock initialized: now 1714701647368163 us; error 26 us; skew 500 ppm
I20240503 02:00:47.369948 13611 webserver.cc:460] Webserver started at http://127.0.99.136:45657/ using document root <none> and password file <none>
I20240503 02:00:47.370445 13611 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:47.370494 13611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:47.370651 13611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:00:47.372256 13611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "558837c679f246dcb3241d66a37d70f1"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.372681 13611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "558837c679f246dcb3241d66a37d70f1"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.375873 13611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:00:47.377982 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.378902 13611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:47.378988 13611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "558837c679f246dcb3241d66a37d70f1"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:47.379079 13611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:47.394394 13611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:47.395021 13611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:47.395174 13611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:47.395690 13611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:47.396572 13611 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:47.396617 13611 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.396668 13611 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:47.396700 13611 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:47.404842 13611 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41447
I20240503 02:00:47.404922 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41447 every 8 connection(s)
I20240503 02:00:47.405752 13611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:47.414019 13739 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:47.414219 13739 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:47.414623 13739 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:47.415171 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 13611
I20240503 02:00:47.415282 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:00:47.415609 12634 ts_manager.cc:194] Registered new tserver with Master: 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:00:47.416229 12634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:45097
I20240503 02:00:47.417536 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:00:47.426182 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:00:47.426244 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 13218
I20240503 02:00:47.432327 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 13349
I20240503 02:00:47.437450 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 13480
I20240503 02:00:47.442255 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 13611
I20240503 02:00:47.541666 13084 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:47.736773 13215 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:51.198683 12634 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:51.218811 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 063a7f2ed28a455697b000f52a4343a3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.219204 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 063a7f2ed28a455697b000f52a4343a3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.219843 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.219538 13017 tablet_service.cc:1467] Processing CreateTablet for tablet da7d3c59eeb949ec84db013f24e656b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.219538 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 063a7f2ed28a455697b000f52a4343a3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.219993 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.220449 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7d3c59eeb949ec84db013f24e656b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.220769 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.221292 13148 tablet_service.cc:1467] Processing CreateTablet for tablet b4b734308d3b42958e3a2830a83d44e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.221508 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b734308d3b42958e3a2830a83d44e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.218811 12754 tablet_service.cc:1467] Processing CreateTablet for tablet b4b734308d3b42958e3a2830a83d44e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.222350 12755 tablet_service.cc:1467] Processing CreateTablet for tablet da7d3c59eeb949ec84db013f24e656b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.222363 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b734308d3b42958e3a2830a83d44e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.222697 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7d3c59eeb949ec84db013f24e656b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.227900 13761 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:51.229574 13760 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:51.229724 13761 tablet_bootstrap.cc:654] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.230530 13761 log.cc:824] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:51.231137 13759 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:51.231676 13760 tablet_bootstrap.cc:654] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.232069 12887 tablet_service.cc:1467] Processing CreateTablet for tablet da7d3c59eeb949ec84db013f24e656b0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.232421 13760 log.cc:824] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:51.232640 13761 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:51.232808 13761 ts_tablet_manager.cc:1367] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:00:51.232956 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7d3c59eeb949ec84db013f24e656b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.233024 12886 tablet_service.cc:1467] Processing CreateTablet for tablet b4b734308d3b42958e3a2830a83d44e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.233361 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b734308d3b42958e3a2830a83d44e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.233968 13760 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:51.234009 13759 tablet_bootstrap.cc:654] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.234189 13760 ts_tablet_manager.cc:1367] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:51.234776 13759 log.cc:824] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:51.236348 13759 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:51.236500 13759 ts_tablet_manager.cc:1367] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:51.237908 13761 raft_consensus.cc:348] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.238097 13761 raft_consensus.cc:374] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.238149 13761 raft_consensus.cc:729] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.238577 13761 consensus_queue.cc:260] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.238826 13760 raft_consensus.cc:348] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.239025 13760 raft_consensus.cc:374] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.239070 13760 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.239428 13761 ts_tablet_manager.cc:1398] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:51.239496 13760 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.239807 13761 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:51.240401 13759 raft_consensus.cc:348] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.240640 13759 raft_consensus.cc:374] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.240754 13761 tablet_bootstrap.cc:654] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.240751 13759 raft_consensus.cc:729] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.241178 13759 consensus_queue.cc:260] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.241845 13761 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:51.241962 13761 ts_tablet_manager.cc:1367] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:51.241978 13760 ts_tablet_manager.cc:1398] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:00:51.242059 13759 ts_tablet_manager.cc:1398] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:51.242292 13760 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:51.242522 13761 raft_consensus.cc:348] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.242645 13761 raft_consensus.cc:374] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.242697 13761 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.242836 13761 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.243237 13761 ts_tablet_manager.cc:1398] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:51.243269 13760 tablet_bootstrap.cc:654] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.243358 13759 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:51.243460 13761 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:51.244377 13761 tablet_bootstrap.cc:654] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.245021 13766 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:51.245427 13759 tablet_bootstrap.cc:654] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.246762 13760 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:51.246871 13760 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:00:51.246951 13759 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:51.247040 13759 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:51.247457 13760 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.247555 13760 raft_consensus.cc:374] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.247592 13760 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.247567 13759 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.247664 13766 tablet_bootstrap.cc:654] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.247676 13759 raft_consensus.cc:374] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.247713 13759 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.247716 13760 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.247851 13759 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.248028 13760 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:51.248111 13759 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.248521 13766 log.cc:824] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:51.249416 13761 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:51.249524 13761 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:51.249928 13761 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.250056 13761 raft_consensus.cc:374] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.250105 13761 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.250231 13761 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.250469 13766 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:51.250496 13761 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.250628 13766 ts_tablet_manager.cc:1367] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:00:51.254084 13766 raft_consensus.cc:348] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.254248 13766 raft_consensus.cc:374] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.254304 13766 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.254673 13766 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.255337 13766 ts_tablet_manager.cc:1398] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:51.255543 13766 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:51.256366 13766 tablet_bootstrap.cc:654] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.257367 13766 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:51.257445 13766 ts_tablet_manager.cc:1367] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:51.257954 13766 raft_consensus.cc:348] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.258052 13766 raft_consensus.cc:374] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.258100 13766 raft_consensus.cc:729] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.258214 13766 consensus_queue.cc:260] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.258436 13766 ts_tablet_manager.cc:1398] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:00:51.282991 13085 tablet.cc:2369] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:51.306653 13775 raft_consensus.cc:482] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.306834 13775 raft_consensus.cc:504] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.307924 13775 leader_election.cc:288] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.312181 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7d3c59eeb949ec84db013f24e656b0" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:51.312252 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7d3c59eeb949ec84db013f24e656b0" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:51.312448 13038 raft_consensus.cc:2454] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:51.312505 12776 raft_consensus.cc:2454] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:51.312871 12841 leader_election.cc:302] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:51.313123 13775 raft_consensus.cc:2781] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.313195 13775 raft_consensus.cc:482] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.313256 13775 raft_consensus.cc:3037] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.314124 13775 raft_consensus.cc:504] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.314437 13775 leader_election.cc:288] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.314774 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7d3c59eeb949ec84db013f24e656b0" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:51.314765 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7d3c59eeb949ec84db013f24e656b0" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:51.314883 13038 raft_consensus.cc:3037] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.314884 12776 raft_consensus.cc:3037] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.316049 13038 raft_consensus.cc:2454] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:51.316362 12843 leader_election.cc:302] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.316398 12776 raft_consensus.cc:2454] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:51.316565 13775 raft_consensus.cc:2781] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.316774 13775 raft_consensus.cc:686] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:51.317091 13775 consensus_queue.cc:237] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.319944 12632 catalog_manager.cc:5484] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
W20240503 02:00:51.332651 12954 tablet.cc:2369] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:00:51.349921 13768 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.350093 13768 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.351131 13768 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:51.354688 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:51.354753 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:51.354812 12776 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.354959 12907 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.355382 13104 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.355613 13768 raft_consensus.cc:2781] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.355690 13768 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.355743 13768 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.356580 13768 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.356936 13768 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:51.357218 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:51.357263 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:51.357321 12907 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.357379 12776 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.358173 12776 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:51.358239 12907 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:51.358475 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:51.358610 13768 raft_consensus.cc:2781] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.358799 13768 raft_consensus.cc:686] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:51.359117 13768 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.361676 12632 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:51.366808 13768 raft_consensus.cc:482] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.366884 13768 raft_consensus.cc:504] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.367244 13768 leader_election.cc:288] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.367574 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:51.367740 12776 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.368009 13103 leader_election.cc:302] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:51.368158 13768 raft_consensus.cc:2781] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.368211 13768 raft_consensus.cc:482] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.368263 13768 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.369007 13768 raft_consensus.cc:504] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.369320 13768 leader_election.cc:288] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.369560 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:51.369675 12776 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.370517 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:51.370543 12776 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:51.370627 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:51.370842 13103 leader_election.cc:302] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:51.370656 13038 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.371021 13768 raft_consensus.cc:2781] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.371083 13768 raft_consensus.cc:686] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:51.371217 13768 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.372553 12632 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:00:51.409276 12823 tablet.cc:2369] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:51.478132 13216 tablet.cc:2369] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:00:51.505439 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:51.505863 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:51.510015 13786 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:00:51.615981 13038 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.616389 13037 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.616773 12776 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.617612 13037 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.617620 13779 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.618489 12907 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.618849 13783 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.621847 13779 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.622291 12776 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.626019 13768 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.626381 13768 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.631222 12776 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:51.631736 13783 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:51.635573 13815 mvcc.cc:205] Tried to move back new op lower bound from 7023417965009006592 to 7023417963796987904. Current Snapshot: MvccSnapshot[applied={T|T < 7023417965009006592}]
I20240503 02:00:51.636216 13822 mvcc.cc:205] Tried to move back new op lower bound from 7023417965009006592 to 7023417963796987904. Current Snapshot: MvccSnapshot[applied={T|T < 7023417965009006592}]
I20240503 02:00:51.640093 13817 mvcc.cc:205] Tried to move back new op lower bound from 7023417965013323776 to 7023417964017778688. Current Snapshot: MvccSnapshot[applied={T|T < 7023417965013323776}]
I20240503 02:00:51.654935 13825 mvcc.cc:205] Tried to move back new op lower bound from 7023417965009006592 to 7023417963796987904. Current Snapshot: MvccSnapshot[applied={T|T < 7023417965009006592}]
INSERT report
rows total: 2000
time total: 153.86 ms
time per row: 0.0769298 ms
I20240503 02:00:51.716660 12632 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:51.723361 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 8108933516894d4190ef9a0cf6063be8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.723459 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 8108933516894d4190ef9a0cf6063be8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.723678 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8108933516894d4190ef9a0cf6063be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.723701 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8108933516894d4190ef9a0cf6063be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.723909 13017 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc67c73560449e81303a948df8f9df (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.724160 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cc67c73560449e81303a948df8f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.726094 13759 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:51.726123 13766 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:51.726987 13759 tablet_bootstrap.cc:654] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.727241 13766 tablet_bootstrap.cc:654] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.728489 12756 tablet_service.cc:1467] Processing CreateTablet for tablet d86b733f38fd4f08bd40865b99e2e7b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.728682 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 8108933516894d4190ef9a0cf6063be8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.728819 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b733f38fd4f08bd40865b99e2e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.728986 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8108933516894d4190ef9a0cf6063be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.731236 13149 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc67c73560449e81303a948df8f9df (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.731309 13760 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:51.731472 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cc67c73560449e81303a948df8f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.732429 13760 tablet_bootstrap.cc:654] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.733711 12886 tablet_service.cc:1467] Processing CreateTablet for tablet d86b733f38fd4f08bd40865b99e2e7b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.733832 13017 tablet_service.cc:1467] Processing CreateTablet for tablet d86b733f38fd4f08bd40865b99e2e7b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.734007 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b733f38fd4f08bd40865b99e2e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.734081 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b733f38fd4f08bd40865b99e2e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.737705 13766 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:51.737811 13766 ts_tablet_manager.cc:1367] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:00:51.738387 13766 raft_consensus.cc:348] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.738518 13766 raft_consensus.cc:374] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.738572 13766 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.739692 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e6cc67c73560449e81303a948df8f9df (DEFAULT_TABLE table=rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:51.740016 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cc67c73560449e81303a948df8f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:51.740351 13759 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:51.740430 13759 ts_tablet_manager.cc:1367] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 02:00:51.740883 13759 raft_consensus.cc:348] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.740983 13759 raft_consensus.cc:374] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.741015 13759 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.741164 13759 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.741441 13759 ts_tablet_manager.cc:1398] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.741607 13759 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:51.742374 13760 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:51.742460 13760 ts_tablet_manager.cc:1367] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:00:51.742511 13759 tablet_bootstrap.cc:654] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.742821 13760 raft_consensus.cc:348] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.742923 13760 raft_consensus.cc:374] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.742954 13760 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.743070 13760 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.743299 13760 ts_tablet_manager.cc:1398] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.743448 13760 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:51.744208 13761 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:51.745018 13761 tablet_bootstrap.cc:654] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.745595 13759 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:51.745682 13759 ts_tablet_manager.cc:1367] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:51.746198 13759 raft_consensus.cc:348] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.746305 13759 raft_consensus.cc:374] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.746343 13759 raft_consensus.cc:729] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.746481 13759 consensus_queue.cc:260] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.746698 13766 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.746794 13759 ts_tablet_manager.cc:1398] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.746980 13759 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:51.747354 13760 tablet_bootstrap.cc:654] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.748368 13759 tablet_bootstrap.cc:654] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.748600 13766 ts_tablet_manager.cc:1398] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240503 02:00:51.748798 13766 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:51.749646 13766 tablet_bootstrap.cc:654] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.750717 13761 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:51.750792 13761 ts_tablet_manager.cc:1367] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:51.750926 13759 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:51.750998 13759 ts_tablet_manager.cc:1367] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:51.751166 13761 raft_consensus.cc:348] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.751248 13761 raft_consensus.cc:374] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.751273 13761 raft_consensus.cc:729] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.751372 13761 consensus_queue.cc:260] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.751428 13759 raft_consensus.cc:348] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.751521 13759 raft_consensus.cc:374] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.751562 13759 raft_consensus.cc:729] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.751593 13761 ts_tablet_manager.cc:1398] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.751677 13759 consensus_queue.cc:260] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.751996 13759 ts_tablet_manager.cc:1398] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.752724 13760 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:51.752799 13760 ts_tablet_manager.cc:1367] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:51.752965 13766 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:51.753058 13766 ts_tablet_manager.cc:1367] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:51.753216 13760 raft_consensus.cc:348] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.753304 13760 raft_consensus.cc:374] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.753329 13760 raft_consensus.cc:729] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.753424 13760 consensus_queue.cc:260] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.753484 13766 raft_consensus.cc:348] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.753584 13766 raft_consensus.cc:374] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.753623 13766 raft_consensus.cc:729] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.753625 13760 ts_tablet_manager.cc:1398] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:51.753741 13766 consensus_queue.cc:260] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.754004 13766 ts_tablet_manager.cc:1398] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:51.754170 13766 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:51.755010 13766 tablet_bootstrap.cc:654] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:51.756053 13766 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:51.756126 13766 ts_tablet_manager.cc:1367] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:51.757072 13766 raft_consensus.cc:348] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.757308 13766 raft_consensus.cc:374] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:51.757376 13766 raft_consensus.cc:729] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:51.758015 13766 consensus_queue.cc:260] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.758339 13766 ts_tablet_manager.cc:1398] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:51.790254 13821 raft_consensus.cc:482] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.790376 13821 raft_consensus.cc:504] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.791312 13821 leader_election.cc:288] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:51.795259 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:51.795413 12907 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:51.795418 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:51.795604 13169 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:51.795770 12973 leader_election.cc:302] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.796002 13821 raft_consensus.cc:2781] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.796067 13821 raft_consensus.cc:482] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.796109 13821 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.797009 13821 raft_consensus.cc:504] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.797331 13821 leader_election.cc:288] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:51.797598 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:51.797710 12907 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.797761 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:51.797868 13169 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.798689 12907 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:51.798848 13169 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:51.798995 12973 leader_election.cc:302] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.799170 13821 raft_consensus.cc:2781] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.799265 13821 raft_consensus.cc:686] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:51.799525 13821 consensus_queue.cc:237] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.801653 12632 catalog_manager.cc:5484] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:51.840143 13783 raft_consensus.cc:482] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.840287 13783 raft_consensus.cc:504] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.840729 13783 leader_election.cc:288] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.841260 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc67c73560449e81303a948df8f9df" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:51.841260 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc67c73560449e81303a948df8f9df" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:51.841435 12907 raft_consensus.cc:2454] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.841435 13038 raft_consensus.cc:2454] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:51.841801 13105 leader_election.cc:302] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:51.841998 13783 raft_consensus.cc:2781] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.842073 13783 raft_consensus.cc:482] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.842115 13783 raft_consensus.cc:3037] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.843031 13783 raft_consensus.cc:504] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.843369 13783 leader_election.cc:288] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:51.843593 13821 raft_consensus.cc:482] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:51.843709 13821 raft_consensus.cc:504] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.843734 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc67c73560449e81303a948df8f9df" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:51.843874 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cc67c73560449e81303a948df8f9df" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:51.843904 12907 raft_consensus.cc:3037] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.843964 13038 raft_consensus.cc:3037] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.844194 13821 leader_election.cc:288] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:51.844576 12906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b733f38fd4f08bd40865b99e2e7b2" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:51.844684 12906 raft_consensus.cc:2454] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:51.844947 13038 raft_consensus.cc:2454] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:51.845068 12973 leader_election.cc:302] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.845129 12907 raft_consensus.cc:2454] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:51.845242 13821 raft_consensus.cc:2781] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:51.845219 13105 leader_election.cc:302] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:51.845280 13821 raft_consensus.cc:482] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:51.845316 13821 raft_consensus.cc:3037] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.845517 13783 raft_consensus.cc:2781] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.845595 13783 raft_consensus.cc:686] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:51.845741 13783 consensus_queue.cc:237] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:51.846189 13821 raft_consensus.cc:504] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.846592 13821 leader_election.cc:288] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:51.846894 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b733f38fd4f08bd40865b99e2e7b2" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:51.847007 12907 raft_consensus.cc:3037] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:51.847796 12907 raft_consensus.cc:2454] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:51.848127 12973 leader_election.cc:302] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:51.848309 13821 raft_consensus.cc:2781] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:51.848318 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b733f38fd4f08bd40865b99e2e7b2" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:51.848340 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86b733f38fd4f08bd40865b99e2e7b2" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:51.848305 12632 catalog_manager.cc:5484] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:51.848431 12776 raft_consensus.cc:2454] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:51.848645 13821 raft_consensus.cc:686] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:51.848798 13821 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:51.850510 12634 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
W20240503 02:00:51.974638 13862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:51.974855 13862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:51.977809 13862 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:00:52.058310 12907 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.058785 13838 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.059491 13038 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.062638 12906 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.062654 12905 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.063591 13821 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.063854 13838 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.063971 13821 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.064832 13169 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.069433 12774 raft_consensus.cc:3037] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.070590 12774 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.071122 13769 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.071480 13769 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.351 ms
time per row: 0.0521753 ms
I20240503 02:00:52.121577 12629 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:52.126863 13149 tablet_service.cc:1467] Processing CreateTablet for tablet c361f82c69ef477d9c3cfd65e07d7f81 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.127063 12756 tablet_service.cc:1467] Processing CreateTablet for tablet c361f82c69ef477d9c3cfd65e07d7f81 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.127184 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c361f82c69ef477d9c3cfd65e07d7f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.127306 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c361f82c69ef477d9c3cfd65e07d7f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.127867 12887 tablet_service.cc:1467] Processing CreateTablet for tablet c361f82c69ef477d9c3cfd65e07d7f81 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.128142 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c361f82c69ef477d9c3cfd65e07d7f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.128388 12754 tablet_service.cc:1467] Processing CreateTablet for tablet f5767bdfcc6249578f7c3ada3c08da6e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.128619 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.129348 13760 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:52.129537 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.129958 13766 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:52.130110 13017 tablet_service.cc:1467] Processing CreateTablet for tablet f5767bdfcc6249578f7c3ada3c08da6e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.130295 13760 tablet_bootstrap.cc:654] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.130379 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.130714 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.130846 13766 tablet_bootstrap.cc:654] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.130944 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.131887 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.131994 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.132133 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.132557 13759 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:52.132826 13761 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:52.133405 13759 tablet_bootstrap.cc:654] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.133690 13761 tablet_bootstrap.cc:654] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.134644 13149 tablet_service.cc:1467] Processing CreateTablet for tablet f5767bdfcc6249578f7c3ada3c08da6e (DEFAULT_TABLE table=rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.134831 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.134841 13761 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:52.134909 13761 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:52.135352 13761 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.135460 13761 raft_consensus.cc:374] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.135493 13761 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.135614 13761 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.135957 13761 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.136118 13761 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:52.136534 13760 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:52.136610 13760 ts_tablet_manager.cc:1367] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:52.136719 13759 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:52.136791 13759 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:52.137001 13761 tablet_bootstrap.cc:654] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.137089 13760 raft_consensus.cc:348] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.137168 13760 raft_consensus.cc:374] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.137189 13760 raft_consensus.cc:729] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.137288 13760 consensus_queue.cc:260] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.137286 13759 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.137387 13759 raft_consensus.cc:374] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.137418 13759 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.137566 13760 ts_tablet_manager.cc:1398] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.137545 13759 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.137719 13760 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:52.137822 13759 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.138496 13760 tablet_bootstrap.cc:654] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.139814 13760 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:52.139911 13760 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:52.140290 13760 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.140403 13760 raft_consensus.cc:374] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.140426 13760 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.140549 13760 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.140913 13760 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.141108 13760 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:52.141510 13766 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:52.141613 13766 ts_tablet_manager.cc:1367] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:00:52.141986 13760 tablet_bootstrap.cc:654] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.142122 13766 raft_consensus.cc:348] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.142216 13766 raft_consensus.cc:374] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.142261 13766 raft_consensus.cc:729] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.142387 13766 consensus_queue.cc:260] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.142719 13766 ts_tablet_manager.cc:1398] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.142892 13766 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:52.142973 13760 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:52.143035 13760 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:52.143437 13760 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.143520 13760 raft_consensus.cc:374] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.143549 13760 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.143657 13760 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.144018 13760 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.144726 13766 tablet_bootstrap.cc:654] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.145596 13761 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:52.145692 13761 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 02:00:52.145740 13766 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:52.145854 13766 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:52.146212 13761 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.146322 13761 raft_consensus.cc:374] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.146364 13761 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.146394 13766 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.146484 13766 raft_consensus.cc:374] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.146525 13766 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.146488 13761 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.146631 13766 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.146876 13766 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.147085 13761 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.147251 13761 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:52.148303 13761 tablet_bootstrap.cc:654] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.149483 13761 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:52.149592 13761 ts_tablet_manager.cc:1367] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:00:52.149933 13761 raft_consensus.cc:348] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.150012 13761 raft_consensus.cc:374] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.150033 13761 raft_consensus.cc:729] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.150156 13761 consensus_queue.cc:260] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.150378 13761 ts_tablet_manager.cc:1398] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.207021 13770 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.207170 13770 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.208429 13770 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:52.212334 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:52.212472 13038 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.212445 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:52.212628 13169 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.212819 12712 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.213074 13770 raft_consensus.cc:2781] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.213132 13770 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.213160 13770 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.214068 13770 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.214387 13770 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:52.214697 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:52.214702 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:52.214804 13169 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.214823 13038 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.215533 13169 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:52.215596 13038 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:52.215850 12711 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.216046 13770 raft_consensus.cc:2781] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.216192 13770 raft_consensus.cc:686] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:52.216468 13770 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.218447 12632 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:00:52.344771 13779 raft_consensus.cc:482] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.344916 13779 raft_consensus.cc:504] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.345386 13779 leader_election.cc:288] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:52.345851 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0613ddd56ed346b3b6dfa9e7c3a4335e" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:52.346010 12774 raft_consensus.cc:2454] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:52.346308 12841 leader_election.cc:302] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.346474 13779 raft_consensus.cc:2781] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.346524 13779 raft_consensus.cc:482] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.346561 13779 raft_consensus.cc:3037] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.347419 13779 raft_consensus.cc:504] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.347774 13779 leader_election.cc:288] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:52.348067 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0613ddd56ed346b3b6dfa9e7c3a4335e" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:52.348188 12774 raft_consensus.cc:3037] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.348908 12774 raft_consensus.cc:2454] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:52.349256 12841 leader_election.cc:302] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.349414 13779 raft_consensus.cc:2781] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.349563 13779 raft_consensus.cc:686] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:52.349725 13779 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.349895 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0613ddd56ed346b3b6dfa9e7c3a4335e" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:52.350030 13168 raft_consensus.cc:3037] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.349875 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0613ddd56ed346b3b6dfa9e7c3a4335e" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:52.350935 13168 raft_consensus.cc:2454] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:52.351402 12632 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:00:52.384579 13770 raft_consensus.cc:482] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.384713 13770 raft_consensus.cc:504] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.385210 13770 leader_election.cc:288] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:52.385710 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c361f82c69ef477d9c3cfd65e07d7f81" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:52.385883 13168 raft_consensus.cc:2454] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.386224 12711 leader_election.cc:302] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.386400 13770 raft_consensus.cc:2781] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.386449 13770 raft_consensus.cc:482] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.386490 13770 raft_consensus.cc:3037] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.387437 13770 raft_consensus.cc:504] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.387809 13770 leader_election.cc:288] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:52.388175 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c361f82c69ef477d9c3cfd65e07d7f81" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:52.388285 13168 raft_consensus.cc:3037] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.389214 13168 raft_consensus.cc:2454] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:52.389207 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c361f82c69ef477d9c3cfd65e07d7f81" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:52.389227 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c361f82c69ef477d9c3cfd65e07d7f81" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:52.389349 12907 raft_consensus.cc:2454] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.389477 12711 leader_election.cc:302] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.389824 13770 raft_consensus.cc:2781] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.389899 13770 raft_consensus.cc:686] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:52.390043 13770 consensus_queue.cc:237] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:52.391474 12632 catalog_manager.cc:5484] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
W20240503 02:00:52.528283 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:52.528512 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:52.531467 13915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:00:52.615231 13168 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.615474 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.615576 13169 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.615856 12905 raft_consensus.cc:3037] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.615953 13908 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.617008 13169 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.617003 12905 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.617393 13779 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.618212 13908 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.618494 13908 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.618760 13908 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:52.618957 12774 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:52.619961 13779 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.581 ms
time per row: 0.0527903 ms
I20240503 02:00:52.677595 12628 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:52.682804 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 34e88e69beb64036bb676b771dabd2a4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.683128 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e88e69beb64036bb676b771dabd2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.683213 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 8292c06377f445849acdc22d08bd612d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.683295 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 8292c06377f445849acdc22d08bd612d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.683472 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.683495 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.683502 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 34e88e69beb64036bb676b771dabd2a4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.683748 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e88e69beb64036bb676b771dabd2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.685602 13975 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:52.685847 13977 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:52.686007 13976 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:52.686553 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 34e88e69beb64036bb676b771dabd2a4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.686630 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 8292c06377f445849acdc22d08bd612d (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.686655 13975 tablet_bootstrap.cc:654] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.686836 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e88e69beb64036bb676b771dabd2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.686970 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.687247 13016 tablet_service.cc:1467] Processing CreateTablet for tablet 097947dce87b409ca4a57cb752fdd5e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.687315 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 097947dce87b409ca4a57cb752fdd5e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.687476 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097947dce87b409ca4a57cb752fdd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.687525 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097947dce87b409ca4a57cb752fdd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.688751 13979 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:52.689250 13976 tablet_bootstrap.cc:654] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.689697 13979 tablet_bootstrap.cc:654] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.690291 13976 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:52.690284 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 097947dce87b409ca4a57cb752fdd5e2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:52.690387 13976 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:52.690505 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097947dce87b409ca4a57cb752fdd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:52.690791 13979 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:52.690866 13979 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:52.690851 13976 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.690949 13976 raft_consensus.cc:374] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.690980 13976 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.691097 13976 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.691248 13979 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.691331 13979 raft_consensus.cc:374] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.691361 13976 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.691361 13979 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.691516 13976 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:52.691480 13979 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.691601 13977 tablet_bootstrap.cc:654] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.692325 13976 tablet_bootstrap.cc:654] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.691733 13979 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.692834 13979 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:52.693038 13977 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:52.693125 13977 ts_tablet_manager.cc:1367] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:52.693420 13979 tablet_bootstrap.cc:654] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.693578 13977 raft_consensus.cc:348] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.693689 13977 raft_consensus.cc:374] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.693725 13977 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.693841 13977 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.694114 13977 ts_tablet_manager.cc:1398] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.695317 13979 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:52.695389 13979 ts_tablet_manager.cc:1367] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:52.695781 13976 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:52.695855 13976 ts_tablet_manager.cc:1367] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:52.695858 13979 raft_consensus.cc:348] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.695941 13979 raft_consensus.cc:374] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.695996 13979 raft_consensus.cc:729] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.696131 13979 consensus_queue.cc:260] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.696300 13976 raft_consensus.cc:348] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.696383 13976 raft_consensus.cc:374] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.696413 13976 raft_consensus.cc:729] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.696470 13979 ts_tablet_manager.cc:1398] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.696534 13976 consensus_queue.cc:260] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.696619 13979 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:52.696784 13976 ts_tablet_manager.cc:1398] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.696940 13977 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:52.697387 13979 tablet_bootstrap.cc:654] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.697959 13977 tablet_bootstrap.cc:654] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.698415 13979 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:52.698501 13979 ts_tablet_manager.cc:1367] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:52.698911 13979 raft_consensus.cc:348] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.699018 13979 raft_consensus.cc:374] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.699059 13979 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.699177 13979 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.699378 13979 ts_tablet_manager.cc:1398] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.699965 13977 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:52.700048 13977 ts_tablet_manager.cc:1367] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:52.700467 13977 raft_consensus.cc:348] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.700569 13977 raft_consensus.cc:374] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.700610 13977 raft_consensus.cc:729] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.700687 13975 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:52.700738 13977 consensus_queue.cc:260] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.700795 13975 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:52.701007 13977 ts_tablet_manager.cc:1398] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.701224 13975 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.701325 13975 raft_consensus.cc:374] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.701355 13975 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.701485 13975 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.701768 13975 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:52.701941 13975 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:52.702801 13975 tablet_bootstrap.cc:654] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:52.703830 13975 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:52.703913 13975 ts_tablet_manager.cc:1367] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:52.704319 13975 raft_consensus.cc:348] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.704417 13975 raft_consensus.cc:374] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:52.704448 13975 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:52.704545 13975 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.704815 13975 ts_tablet_manager.cc:1398] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:52.752204 13821 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.752329 13821 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.752696 13821 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:52.753054 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:52.753044 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:52.753191 12905 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:52.753191 12774 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:52.753448 12973 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:52.753600 13821 raft_consensus.cc:2781] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.753659 13821 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.753707 13821 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.754456 13821 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.754753 13821 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:52.755046 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:52.755154 12774 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.755182 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:52.755299 12905 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.756150 12774 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:52.756251 12905 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:52.756440 12972 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.756610 13821 raft_consensus.cc:2781] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.756757 13821 raft_consensus.cc:686] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:52.756875 13821 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:52.758507 12634 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:00:52.849473 13948 raft_consensus.cc:482] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.849609 13948 raft_consensus.cc:504] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.850085 13948 leader_election.cc:288] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:52.850504 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:52.850505 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:52.850646 13168 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.850649 13038 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:52.850955 12712 leader_election.cc:302] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.851109 13948 raft_consensus.cc:2781] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.851166 13948 raft_consensus.cc:482] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.851213 13948 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.852083 13948 raft_consensus.cc:504] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.852381 13948 leader_election.cc:288] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:52.852696 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:52.852712 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:52.852798 13168 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.852806 13038 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.853492 13038 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:52.853498 13168 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:52.853742 12711 leader_election.cc:302] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:52.853931 13948 raft_consensus.cc:2781] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.854028 13948 raft_consensus.cc:686] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:52.854211 13948 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:52.855744 12634 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:00:52.892215 13954 raft_consensus.cc:482] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:52.892351 13954 raft_consensus.cc:504] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.892755 13954 leader_election.cc:288] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:52.893119 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097947dce87b409ca4a57cb752fdd5e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:52.893121 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097947dce87b409ca4a57cb752fdd5e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:52.893245 12905 raft_consensus.cc:2454] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:52.893256 13038 raft_consensus.cc:2454] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:52.893535 13104 leader_election.cc:302] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:52.893721 13954 raft_consensus.cc:2781] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:52.893772 13954 raft_consensus.cc:482] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:52.893798 13954 raft_consensus.cc:3037] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.894578 13954 raft_consensus.cc:504] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.894903 13954 leader_election.cc:288] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:52.895203 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097947dce87b409ca4a57cb752fdd5e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:52.895234 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097947dce87b409ca4a57cb752fdd5e2" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:52.895320 13038 raft_consensus.cc:3037] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.895330 12905 raft_consensus.cc:3037] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:52.896066 13038 raft_consensus.cc:2454] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:52.896088 12905 raft_consensus.cc:2454] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:52.896350 13105 leader_election.cc:302] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:52.896517 13954 raft_consensus.cc:2781] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:52.896589 13954 raft_consensus.cc:686] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:52.896760 13954 consensus_queue.cc:237] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:52.898190 12634 catalog_manager.cc:5484] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:00:53.026800 13992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:53.027063 13992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:53.030057 13992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:00:53.131466 13038 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.132515 12905 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.132845 12774 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.133260 13168 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.133622 13821 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.133908 13821 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.134218 13954 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.135612 13948 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.138900 13038 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.144155 13949 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.146651 12907 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.147092 13954 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 133.735 ms
time per row: 0.0668677 ms
I20240503 02:00:53.210390 12630 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:53.215652 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 3b829f779c8a459597c47dc3e42a4c46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.215811 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 3b829f779c8a459597c47dc3e42a4c46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.216032 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b829f779c8a459597c47dc3e42a4c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.216100 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b829f779c8a459597c47dc3e42a4c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.216593 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 490776bd408a48a9bcbbc7a501baa8ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.216826 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.217001 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 0633596226954c439d2f671c25eebfe6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.217216 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0633596226954c439d2f671c25eebfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.218479 14042 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:53.218436 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 3b829f779c8a459597c47dc3e42a4c46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.218712 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b829f779c8a459597c47dc3e42a4c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.219503 14042 tablet_bootstrap.cc:654] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.219825 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 490776bd408a48a9bcbbc7a501baa8ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.219902 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 0633596226954c439d2f671c25eebfe6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.220074 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.220319 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0633596226954c439d2f671c25eebfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.221065 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 490776bd408a48a9bcbbc7a501baa8ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.221274 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.222581 14045 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:53.222668 14042 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:53.222877 14042 ts_tablet_manager.cc:1367] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:00:53.223342 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 0633596226954c439d2f671c25eebfe6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.223443 14045 tablet_bootstrap.cc:654] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.223562 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0633596226954c439d2f671c25eebfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.223579 14042 raft_consensus.cc:348] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.223696 14042 raft_consensus.cc:374] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.223788 14042 raft_consensus.cc:729] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.223956 14042 consensus_queue.cc:260] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.224072 14041 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:53.224334 14042 ts_tablet_manager.cc:1398] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:53.224520 14042 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:53.224993 14041 tablet_bootstrap.cc:654] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.225498 14042 tablet_bootstrap.cc:654] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.225662 14043 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:53.225986 14045 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:53.226078 14045 ts_tablet_manager.cc:1367] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:53.226491 14043 tablet_bootstrap.cc:654] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.226508 14045 raft_consensus.cc:348] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.226622 14045 raft_consensus.cc:374] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.226672 14045 raft_consensus.cc:729] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.226830 14045 consensus_queue.cc:260] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.227450 14043 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:53.227537 14043 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.227656 14045 ts_tablet_manager.cc:1398] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.227825 14045 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:53.227998 14043 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.228086 14043 raft_consensus.cc:374] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.228119 14043 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.228237 14043 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.228489 14043 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.228760 14043 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:53.228967 14041 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:53.229045 14041 ts_tablet_manager.cc:1367] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:53.229351 14045 tablet_bootstrap.cc:654] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.229591 14043 tablet_bootstrap.cc:654] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.229518 14041 raft_consensus.cc:348] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.229627 14041 raft_consensus.cc:374] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.229662 14041 raft_consensus.cc:729] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.229810 14041 consensus_queue.cc:260] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.230106 14041 ts_tablet_manager.cc:1398] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.230271 14041 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:53.230748 14045 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:53.230832 14045 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:53.230959 14043 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:53.231031 14043 ts_tablet_manager.cc:1367] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.231175 14041 tablet_bootstrap.cc:654] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.231232 14045 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.231323 14045 raft_consensus.cc:374] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.231357 14045 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.231417 14043 raft_consensus.cc:348] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.231523 14043 raft_consensus.cc:374] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.231562 14043 raft_consensus.cc:729] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.231642 14045 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.231664 14043 consensus_queue.cc:260] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.232025 14043 ts_tablet_manager.cc:1398] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.232440 14045 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:53.233575 14041 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:53.233700 14042 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:53.233718 14041 ts_tablet_manager.cc:1367] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:53.233845 14042 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:00:53.234330 14041 raft_consensus.cc:348] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.234339 14042 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.234417 14041 raft_consensus.cc:374] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.234437 14042 raft_consensus.cc:374] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.234452 14041 raft_consensus.cc:729] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.234510 14042 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.234588 14041 consensus_queue.cc:260] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.234658 14042 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.234997 14042 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:53.235181 14042 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:53.235234 14041 ts_tablet_manager.cc:1398] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.236141 14042 tablet_bootstrap.cc:654] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.237237 14042 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:53.237318 14042 ts_tablet_manager.cc:1367] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.237746 14042 raft_consensus.cc:348] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.237850 14042 raft_consensus.cc:374] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.237882 14042 raft_consensus.cc:729] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.238049 14042 consensus_queue.cc:260] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.238299 14042 ts_tablet_manager.cc:1398] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.262790 13911 raft_consensus.cc:482] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:53.262904 13911 raft_consensus.cc:504] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.263281 13911 leader_election.cc:288] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:53.263620 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490776bd408a48a9bcbbc7a501baa8ce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:53.263653 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490776bd408a48a9bcbbc7a501baa8ce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:53.263794 12774 raft_consensus.cc:2454] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:53.263797 13168 raft_consensus.cc:2454] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:53.264106 12842 leader_election.cc:302] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:53.264266 13911 raft_consensus.cc:2781] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:53.264321 13911 raft_consensus.cc:482] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:53.264348 13911 raft_consensus.cc:3037] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.265172 13911 raft_consensus.cc:504] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.265484 13911 leader_election.cc:288] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:53.265810 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490776bd408a48a9bcbbc7a501baa8ce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:53.265841 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490776bd408a48a9bcbbc7a501baa8ce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:53.265909 12774 raft_consensus.cc:3037] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.265959 13168 raft_consensus.cc:3037] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.266638 12774 raft_consensus.cc:2454] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:53.266697 13168 raft_consensus.cc:2454] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:53.266885 12841 leader_election.cc:302] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:53.267046 13911 raft_consensus.cc:2781] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:53.267210 13911 raft_consensus.cc:686] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:53.267344 13911 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.268908 12631 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:53.356956 13948 raft_consensus.cc:482] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:53.357091 13948 raft_consensus.cc:504] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.357509 13948 leader_election.cc:288] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:53.357980 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b829f779c8a459597c47dc3e42a4c46" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:53.358022 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b829f779c8a459597c47dc3e42a4c46" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:53.358140 12907 raft_consensus.cc:2454] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:53.358152 13038 raft_consensus.cc:2454] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:53.358453 12712 leader_election.cc:302] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:53.358635 13948 raft_consensus.cc:2781] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:53.358703 13948 raft_consensus.cc:482] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:53.358758 13948 raft_consensus.cc:3037] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.359555 13948 raft_consensus.cc:504] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.359890 13948 leader_election.cc:288] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:53.360185 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b829f779c8a459597c47dc3e42a4c46" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:53.360198 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b829f779c8a459597c47dc3e42a4c46" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:53.360296 13038 raft_consensus.cc:3037] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.360311 12907 raft_consensus.cc:3037] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.361021 12907 raft_consensus.cc:2454] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:53.361037 13038 raft_consensus.cc:2454] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:53.361270 12711 leader_election.cc:302] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:53.361423 13948 raft_consensus.cc:2781] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:53.361508 13948 raft_consensus.cc:686] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:53.361660 13948 consensus_queue.cc:237] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.363274 12631 catalog_manager.cc:5484] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:00:53.479250 13838 raft_consensus.cc:482] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:53.479383 13838 raft_consensus.cc:504] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.479847 13838 leader_election.cc:288] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:53.480301 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0633596226954c439d2f671c25eebfe6" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:53.480301 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0633596226954c439d2f671c25eebfe6" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:53.480443 12774 raft_consensus.cc:2454] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:53.480443 13038 raft_consensus.cc:2454] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:53.480731 13105 leader_election.cc:302] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:53.480918 13838 raft_consensus.cc:2781] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:53.480980 13838 raft_consensus.cc:482] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:53.481010 13838 raft_consensus.cc:3037] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.481827 13838 raft_consensus.cc:504] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.482157 13838 leader_election.cc:288] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:53.482461 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0633596226954c439d2f671c25eebfe6" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:53.482483 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0633596226954c439d2f671c25eebfe6" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:53.482569 13038 raft_consensus.cc:3037] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.482590 12774 raft_consensus.cc:3037] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.483270 13038 raft_consensus.cc:2454] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:53.483384 12774 raft_consensus.cc:2454] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:53.483557 13105 leader_election.cc:302] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:53.483738 13838 raft_consensus.cc:2781] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:53.483830 13838 raft_consensus.cc:686] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:53.483984 13838 consensus_queue.cc:237] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.485584 12631 catalog_manager.cc:5484] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:00:53.619663 14058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:53.619925 14058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:53.623046 14058 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:00:53.709498 12774 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.709553 13038 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.709861 13168 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.709995 13838 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.710494 13911 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.711575 13954 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.714156 12776 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.714812 13911 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.717779 12907 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.719666 13949 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.736117 13038 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:53.736565 13949 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:53.761065 14094 mvcc.cc:205] Tried to move back new op lower bound from 7023417973622460416 to 7023417972170612736. Current Snapshot: MvccSnapshot[applied={T|T < 7023417973622460416}]
INSERT report
rows total: 2000
time total: 120.331 ms
time per row: 0.0601654 ms
I20240503 02:00:53.786337 12631 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:53.792121 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 141097686fe24c8f9956dcc1ad1d253c (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.792414 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.792924 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 141097686fe24c8f9956dcc1ad1d253c (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.793013 13018 tablet_service.cc:1467] Processing CreateTablet for tablet b39fa1cb3863414287a47a0832c8b55d (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.793183 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.793314 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39fa1cb3863414287a47a0832c8b55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.793632 12756 tablet_service.cc:1467] Processing CreateTablet for tablet b39fa1cb3863414287a47a0832c8b55d (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.793897 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39fa1cb3863414287a47a0832c8b55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.794775 14118 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:53.795189 13016 tablet_service.cc:1467] Processing CreateTablet for tablet 613074d3c4514bd5b64a06c59af09a56 (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.795413 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.795648 14118 tablet_bootstrap.cc:654] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.796048 14120 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:53.796748 14118 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:53.796842 14118 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.797178 14120 tablet_bootstrap.cc:654] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.797431 12885 tablet_service.cc:1467] Processing CreateTablet for tablet b39fa1cb3863414287a47a0832c8b55d (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.797458 14119 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:53.797396 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 141097686fe24c8f9956dcc1ad1d253c (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.797580 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 613074d3c4514bd5b64a06c59af09a56 (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.797698 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.797809 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.797971 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39fa1cb3863414287a47a0832c8b55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.798240 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 613074d3c4514bd5b64a06c59af09a56 (DEFAULT_TABLE table=rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:53.798466 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:53.799036 14119 tablet_bootstrap.cc:654] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.800263 14119 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:53.800354 14119 ts_tablet_manager.cc:1367] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:53.800431 14123 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:53.800805 14119 raft_consensus.cc:348] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.800923 14119 raft_consensus.cc:374] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.800962 14119 raft_consensus.cc:729] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.801097 14119 consensus_queue.cc:260] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.801447 14119 ts_tablet_manager.cc:1398] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.801612 14119 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:53.801828 14118 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.801949 14118 raft_consensus.cc:374] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.801991 14118 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.802115 14118 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.802392 14118 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:53.802539 14119 tablet_bootstrap.cc:654] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.802659 14123 tablet_bootstrap.cc:654] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.802861 14118 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:53.803683 14118 tablet_bootstrap.cc:654] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.803833 14120 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:53.803970 14120 ts_tablet_manager.cc:1367] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:53.804033 14119 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:53.804111 14119 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:53.804514 14119 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.804611 14119 raft_consensus.cc:374] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.804652 14119 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.804782 14119 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.804946 14118 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:53.805017 14118 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:53.805104 14119 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.805256 14119 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:53.805462 14118 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.805553 14118 raft_consensus.cc:374] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.805585 14118 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.805707 14118 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.806036 14118 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:53.806494 14120 raft_consensus.cc:348] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.806603 14120 raft_consensus.cc:374] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.806641 14120 raft_consensus.cc:729] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.806864 14119 tablet_bootstrap.cc:654] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.806810 14120 consensus_queue.cc:260] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.807196 14120 ts_tablet_manager.cc:1398] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:53.807876 14119 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:53.807953 14119 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:53.808352 14119 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.808418 14123 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:53.808456 14119 raft_consensus.cc:374] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.808490 14119 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.808495 14123 ts_tablet_manager.cc:1367] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:00:53.808600 14119 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.808856 14119 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:53.808967 14123 raft_consensus.cc:348] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.809062 14123 raft_consensus.cc:374] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.809114 14123 raft_consensus.cc:729] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.809253 14123 consensus_queue.cc:260] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.809528 14123 ts_tablet_manager.cc:1398] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:53.809705 14123 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:53.810582 14123 tablet_bootstrap.cc:654] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.811625 14123 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:53.811707 14123 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:53.812155 14123 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.812278 14123 raft_consensus.cc:374] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.812316 14123 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.812434 14123 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.812760 14123 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:53.812918 14123 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:53.813711 14123 tablet_bootstrap.cc:654] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:53.814668 14123 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:53.814738 14123 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:53.815109 14123 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.815209 14123 raft_consensus.cc:374] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:53.815244 14123 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:53.815357 14123 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:53.815603 14123 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:53.905531 13838 raft_consensus.cc:482] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:53.905664 13838 raft_consensus.cc:504] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.906160 13838 leader_election.cc:288] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:53.906522 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141097686fe24c8f9956dcc1ad1d253c" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:53.906525 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141097686fe24c8f9956dcc1ad1d253c" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:53.906662 12907 raft_consensus.cc:2454] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:53.906672 13038 raft_consensus.cc:2454] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:53.906968 13104 leader_election.cc:302] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:53.907150 13838 raft_consensus.cc:2781] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:53.907219 13838 raft_consensus.cc:482] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:53.907269 13838 raft_consensus.cc:3037] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.909853 13838 raft_consensus.cc:504] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.910216 13838 leader_election.cc:288] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:53.910522 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141097686fe24c8f9956dcc1ad1d253c" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:53.910636 13038 raft_consensus.cc:3037] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.911507 13038 raft_consensus.cc:2454] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:53.911731 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141097686fe24c8f9956dcc1ad1d253c" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:53.911808 13105 leader_election.cc:302] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:53.911852 12907 raft_consensus.cc:3037] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.911973 13838 raft_consensus.cc:2781] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:53.912045 13838 raft_consensus.cc:686] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:53.912178 13838 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:53.912667 12907 raft_consensus.cc:2454] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:53.913934 12634 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:00:53.953812 13911 raft_consensus.cc:482] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:53.953972 13911 raft_consensus.cc:504] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.954375 13911 leader_election.cc:288] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:53.954792 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39fa1cb3863414287a47a0832c8b55d" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:53.954792 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39fa1cb3863414287a47a0832c8b55d" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:53.954924 12774 raft_consensus.cc:2454] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:53.954936 13038 raft_consensus.cc:2454] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:53.955230 12843 leader_election.cc:302] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:53.955392 13911 raft_consensus.cc:2781] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:53.955443 13911 raft_consensus.cc:482] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:53.955471 13911 raft_consensus.cc:3037] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.956317 13911 raft_consensus.cc:504] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.956610 13911 leader_election.cc:288] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:53.956943 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39fa1cb3863414287a47a0832c8b55d" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:53.956991 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39fa1cb3863414287a47a0832c8b55d" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:53.957057 13038 raft_consensus.cc:3037] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.957089 12774 raft_consensus.cc:3037] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:53.957954 13038 raft_consensus.cc:2454] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:53.957954 12774 raft_consensus.cc:2454] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:53.958243 12843 leader_election.cc:302] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:53.958428 13911 raft_consensus.cc:2781] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:53.958523 13911 raft_consensus.cc:686] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:53.958673 13911 consensus_queue.cc:237] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:53.960402 12634 catalog_manager.cc:5484] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:00:54.059111 13988 raft_consensus.cc:482] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.059237 13988 raft_consensus.cc:504] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.059613 13988 leader_election.cc:288] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.060047 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613074d3c4514bd5b64a06c59af09a56" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:54.060134 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613074d3c4514bd5b64a06c59af09a56" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:54.060205 12907 raft_consensus.cc:2454] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:54.060268 13168 raft_consensus.cc:2454] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:54.060495 12973 leader_election.cc:302] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:54.060709 13988 raft_consensus.cc:2781] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:54.060779 13988 raft_consensus.cc:482] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:54.060832 13988 raft_consensus.cc:3037] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.061661 13988 raft_consensus.cc:504] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.061949 13988 leader_election.cc:288] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.062230 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613074d3c4514bd5b64a06c59af09a56" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:54.062305 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613074d3c4514bd5b64a06c59af09a56" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:54.062345 12907 raft_consensus.cc:3037] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.062408 13168 raft_consensus.cc:3037] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.063205 13168 raft_consensus.cc:2454] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:54.063447 12973 leader_election.cc:302] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:54.063593 13988 raft_consensus.cc:2781] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:54.063834 13988 raft_consensus.cc:686] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:54.063993 13988 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.065798 12634 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:54.080520 12907 raft_consensus.cc:2454] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
W20240503 02:00:54.195312 14136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:54.195559 14136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:54.198761 14136 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 02:00:54.288096 12907 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.288166 13038 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.288707 13168 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.288794 14174 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.288918 13838 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.289191 12907 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.289600 13988 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.289873 13988 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.301975 13036 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.302485 13911 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.302940 14180 mvcc.cc:205] Tried to move back new op lower bound from 7023417975962488832 to 7023417975047729152. Current Snapshot: MvccSnapshot[applied={T|T < 7023417975962488832}]
I20240503 02:00:54.305279 12774 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.306021 13911 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.311254 14175 mvcc.cc:205] Tried to move back new op lower bound from 7023417975962488832 to 7023417975047729152. Current Snapshot: MvccSnapshot[applied={T|T < 7023417975962488832}]
I20240503 02:00:54.326509 14189 mvcc.cc:205] Tried to move back new op lower bound from 7023417976014704640 to 7023417974615789568. Current Snapshot: MvccSnapshot[applied={T|T < 7023417976014704640}]
INSERT report
rows total: 2000
time total: 120.517 ms
time per row: 0.0602586 ms
I20240503 02:00:54.362222 12630 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:54.367516 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 13a95f88ac1c459ba01be0743297f696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.367576 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 13a95f88ac1c459ba01be0743297f696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.367866 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.367913 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.367966 13016 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad545d33bbc40458c75ed7c90d92ba7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.368050 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad545d33bbc40458c75ed7c90d92ba7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.368206 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad545d33bbc40458c75ed7c90d92ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.368294 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad545d33bbc40458c75ed7c90d92ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.370055 14199 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:54.370369 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 13a95f88ac1c459ba01be0743297f696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.370481 12755 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad545d33bbc40458c75ed7c90d92ba7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.370594 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.370711 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad545d33bbc40458c75ed7c90d92ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.370996 14199 tablet_bootstrap.cc:654] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.372354 12754 tablet_service.cc:1467] Processing CreateTablet for tablet 007feed148fb44e9a6fe9d449b6d8f9d (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.372428 14199 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:54.372496 14199 ts_tablet_manager.cc:1367] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:54.372570 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007feed148fb44e9a6fe9d449b6d8f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.372778 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 007feed148fb44e9a6fe9d449b6d8f9d (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.372900 14200 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:54.373008 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007feed148fb44e9a6fe9d449b6d8f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.372948 14199 raft_consensus.cc:348] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.373051 14199 raft_consensus.cc:374] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.373090 14199 raft_consensus.cc:729] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.373204 14199 consensus_queue.cc:260] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.373477 14199 ts_tablet_manager.cc:1398] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.373634 14199 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:54.374096 13149 tablet_service.cc:1467] Processing CreateTablet for tablet 007feed148fb44e9a6fe9d449b6d8f9d (DEFAULT_TABLE table=rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.374153 14200 tablet_bootstrap.cc:654] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.374300 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007feed148fb44e9a6fe9d449b6d8f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.374468 14199 tablet_bootstrap.cc:654] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.375129 14200 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:54.375200 14200 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:54.375532 14200 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.375638 14200 raft_consensus.cc:374] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.375671 14200 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.375804 14200 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.376009 14199 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:54.376080 14200 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.376093 14199 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:54.376261 14200 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:54.376663 14199 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.376762 14199 raft_consensus.cc:374] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.376801 14199 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.376932 14199 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.377101 14200 tablet_bootstrap.cc:654] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.377233 14199 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.377265 14201 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:54.377404 14199 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:54.378276 14199 tablet_bootstrap.cc:654] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.378281 14201 tablet_bootstrap.cc:654] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.378540 14200 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:54.378615 14200 ts_tablet_manager.cc:1367] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:54.379225 14200 raft_consensus.cc:348] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.379321 14200 raft_consensus.cc:374] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.379349 14200 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.379495 14201 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:54.379493 14200 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.379578 14201 ts_tablet_manager.cc:1367] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:54.379848 14199 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:54.379848 14200 ts_tablet_manager.cc:1398] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:54.379930 14199 ts_tablet_manager.cc:1367] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:54.380085 14201 raft_consensus.cc:348] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.380175 14201 raft_consensus.cc:374] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.380205 14201 raft_consensus.cc:729] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.380324 14201 consensus_queue.cc:260] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.380331 14199 raft_consensus.cc:348] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.380406 14199 raft_consensus.cc:374] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.380439 14199 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.380532 14199 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.380622 14201 ts_tablet_manager.cc:1398] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.380766 14199 ts_tablet_manager.cc:1398] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.389613 14208 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:54.390367 14208 tablet_bootstrap.cc:654] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.391244 14208 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:54.391327 14208 ts_tablet_manager.cc:1367] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:54.391659 14208 raft_consensus.cc:348] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.391775 14208 raft_consensus.cc:374] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.391816 14208 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.391947 14208 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.392174 14208 ts_tablet_manager.cc:1398] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.392333 14208 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:54.392963 14208 tablet_bootstrap.cc:654] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.393957 14208 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:54.394044 14208 ts_tablet_manager.cc:1367] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:54.394390 14208 raft_consensus.cc:348] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.394502 14208 raft_consensus.cc:374] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.394546 14208 raft_consensus.cc:729] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.394682 14208 consensus_queue.cc:260] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.394981 14208 ts_tablet_manager.cc:1398] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.395130 14208 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:54.395853 14208 tablet_bootstrap.cc:654] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.396783 14208 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:54.396853 14208 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:54.397306 14208 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.397383 14208 raft_consensus.cc:374] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.397413 14208 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.397490 14208 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.397657 14208 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:54.516796 13949 raft_consensus.cc:482] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.516994 13949 raft_consensus.cc:504] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.517478 13949 leader_election.cc:288] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:54.517839 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad545d33bbc40458c75ed7c90d92ba7" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:54.517906 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad545d33bbc40458c75ed7c90d92ba7" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:54.518010 13168 raft_consensus.cc:2454] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.518076 13038 raft_consensus.cc:2454] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.518337 12711 leader_election.cc:302] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.518491 13949 raft_consensus.cc:2781] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:54.518553 13949 raft_consensus.cc:482] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:54.518579 13949 raft_consensus.cc:3037] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.519479 13949 raft_consensus.cc:504] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.519901 13949 leader_election.cc:288] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:54.520236 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad545d33bbc40458c75ed7c90d92ba7" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:54.520363 13038 raft_consensus.cc:3037] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.520537 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad545d33bbc40458c75ed7c90d92ba7" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:54.520684 13168 raft_consensus.cc:3037] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.521389 13038 raft_consensus.cc:2454] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.521667 12712 leader_election.cc:302] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.521817 13168 raft_consensus.cc:2454] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.521874 13949 raft_consensus.cc:2781] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:54.522118 13949 raft_consensus.cc:686] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:54.522282 13949 consensus_queue.cc:237] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:54.524322 12628 catalog_manager.cc:5484] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:00:54.598791 13949 raft_consensus.cc:482] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.598930 13949 raft_consensus.cc:504] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.599332 13949 leader_election.cc:288] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.599738 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:54.599854 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:54.599921 12907 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.600003 13168 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.600250 12711 leader_election.cc:302] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.600423 13949 raft_consensus.cc:2781] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:54.600486 13949 raft_consensus.cc:482] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:54.600526 13949 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.601296 13949 raft_consensus.cc:504] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.601578 13949 leader_election.cc:288] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.601908 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:54.602016 12907 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.602010 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:54.602144 13167 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.602778 12907 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.602975 13167 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.603050 12711 leader_election.cc:302] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.603195 13949 raft_consensus.cc:2781] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:54.603289 13949 raft_consensus.cc:686] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:54.603425 13949 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.605216 12628 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:00:54.645807 13949 raft_consensus.cc:482] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.645946 13949 raft_consensus.cc:504] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.646363 13949 leader_election.cc:288] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:54.646726 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:54.646795 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:54.646865 13167 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.646942 12907 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:54.647152 12711 leader_election.cc:302] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.647328 13949 raft_consensus.cc:2781] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:54.647377 13949 raft_consensus.cc:482] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:54.647415 13949 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.648243 13949 raft_consensus.cc:504] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.648540 13949 leader_election.cc:288] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:54.648885 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:54.648937 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:54.649003 13167 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.649050 12907 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.649731 13167 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.649739 12907 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:54.650002 12711 leader_election.cc:302] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:54.650147 13949 raft_consensus.cc:2781] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:54.650233 13949 raft_consensus.cc:686] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:54.650362 13949 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.652156 12628 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
W20240503 02:00:54.781047 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:54.781275 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:54.784348 14216 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:00:54.858719 13167 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.859535 13949 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.860880 12907 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.860937 12905 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.861269 13038 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.861377 13949 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.861482 14212 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.865226 13168 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.865533 13169 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:54.865721 14246 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.866255 14212 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:54.870750 14212 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.63 ms
time per row: 0.0518149 ms
I20240503 02:00:54.930320 12630 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:54.935529 13016 tablet_service.cc:1467] Processing CreateTablet for tablet 79c5f35eb8b546de8015c5235913efce (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.935529 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 79c5f35eb8b546de8015c5235913efce (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.935871 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79c5f35eb8b546de8015c5235913efce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.935870 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79c5f35eb8b546de8015c5235913efce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.936326 13017 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.936548 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.936548 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 79c5f35eb8b546de8015c5235913efce (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.936801 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79c5f35eb8b546de8015c5235913efce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.938143 12754 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.938378 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.938591 14276 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:54.939602 14276 tablet_bootstrap.cc:654] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.939589 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 86e314466c01488bb661c596c405db51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.939611 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 86e314466c01488bb661c596c405db51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.939821 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.939939 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.940758 13149 tablet_service.cc:1467] Processing CreateTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.940758 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 86e314466c01488bb661c596c405db51 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:54.941087 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.941210 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:54.942039 14277 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:54.942795 14275 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:54.942899 14277 tablet_bootstrap.cc:654] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.943641 14275 tablet_bootstrap.cc:654] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.943660 14276 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:54.943737 14276 ts_tablet_manager.cc:1367] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:54.943962 14277 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:54.943974 14279 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:54.944038 14277 ts_tablet_manager.cc:1367] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:54.944209 14276 raft_consensus.cc:348] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.944304 14276 raft_consensus.cc:374] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.944334 14276 raft_consensus.cc:729] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.944407 14277 raft_consensus.cc:348] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.944489 14277 raft_consensus.cc:374] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.944451 14276 consensus_queue.cc:260] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.944517 14277 raft_consensus.cc:729] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.944542 14275 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:54.944608 14275 ts_tablet_manager.cc:1367] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:54.944622 14277 consensus_queue.cc:260] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.944734 14276 ts_tablet_manager.cc:1398] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:54.944891 14276 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:54.944875 14277 ts_tablet_manager.cc:1398] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.945021 14277 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:54.945017 14275 raft_consensus.cc:348] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.945106 14275 raft_consensus.cc:374] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.945134 14275 raft_consensus.cc:729] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.945256 14275 consensus_queue.cc:260] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.944792 14279 tablet_bootstrap.cc:654] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.945524 14275 ts_tablet_manager.cc:1398] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.945710 14275 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:54.945748 14276 tablet_bootstrap.cc:654] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.945802 14277 tablet_bootstrap.cc:654] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.946480 14279 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:54.946542 14275 tablet_bootstrap.cc:654] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.946576 14279 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:54.947049 14279 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.947170 14279 raft_consensus.cc:374] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.947201 14279 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.947314 14279 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.947556 14279 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.947737 14279 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:54.948618 14279 tablet_bootstrap.cc:654] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.948776 14277 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:54.948844 14277 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:54.949258 14277 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.949362 14277 raft_consensus.cc:374] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.949402 14277 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.949554 14275 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:54.949635 14275 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:54.949735 14276 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:54.949708 14277 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.949808 14276 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:54.950062 14275 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.950103 14277 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.950155 14275 raft_consensus.cc:374] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.950187 14275 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.950217 14276 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.950299 14276 raft_consensus.cc:374] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.950330 14276 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.950299 14275 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.950446 14276 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.950557 14275 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.950695 14275 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:54.950716 14276 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.951545 14275 tablet_bootstrap.cc:654] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:54.952620 14279 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:54.952719 14279 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:00:54.953168 14279 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.953263 14279 raft_consensus.cc:374] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.953296 14279 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.953393 14275 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:54.953467 14275 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:54.953667 14279 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.954001 14279 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.954077 14275 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.954173 14275 raft_consensus.cc:374] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:54.954205 14275 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:54.954344 14275 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.954579 14275 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:54.982196 14134 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.982316 14134 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.983050 14134 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.983371 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:54.983455 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:54.983505 12907 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:54.983593 13167 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:54.983843 12973 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:54.984030 14134 raft_consensus.cc:2781] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:54.984086 14134 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:54.984112 14134 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.984968 14134 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.985256 14134 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:54.985544 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:54.985657 12907 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.985631 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:54.985764 13167 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:54.986418 12907 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:54.986542 13167 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:54.986882 12973 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:54.987087 14134 raft_consensus.cc:2781] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:54.987287 14134 raft_consensus.cc:686] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:54.987422 14134 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:54.989017 12625 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:00:54.999091 14260 raft_consensus.cc:482] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:54.999195 14260 raft_consensus.cc:504] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:54.999540 14260 leader_election.cc:288] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:54.999899 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c5f35eb8b546de8015c5235913efce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:54.999930 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c5f35eb8b546de8015c5235913efce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:55.000008 13038 raft_consensus.cc:2454] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:55.000061 12774 raft_consensus.cc:2454] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:55.000306 12843 leader_election.cc:302] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:55.000504 14260 raft_consensus.cc:2781] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:55.000560 14260 raft_consensus.cc:482] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:55.000613 14260 raft_consensus.cc:3037] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.001355 14260 raft_consensus.cc:504] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.001636 14260 leader_election.cc:288] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:55.001942 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c5f35eb8b546de8015c5235913efce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:55.001942 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79c5f35eb8b546de8015c5235913efce" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:55.002048 13038 raft_consensus.cc:3037] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.002058 12774 raft_consensus.cc:3037] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.002910 12774 raft_consensus.cc:2454] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:55.002907 13038 raft_consensus.cc:2454] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:55.003161 12841 leader_election.cc:302] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.003337 14260 raft_consensus.cc:2781] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:55.003417 14260 raft_consensus.cc:686] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:55.003551 14260 consensus_queue.cc:237] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.005174 12625 catalog_manager.cc:5484] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:00:55.158792 14134 raft_consensus.cc:482] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:55.158947 14134 raft_consensus.cc:504] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.159390 14134 leader_election.cc:288] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:55.159826 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:55.159826 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:55.159969 12774 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:55.159981 13167 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:55.160245 12972 leader_election.cc:302] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.160398 14134 raft_consensus.cc:2781] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:55.160459 14134 raft_consensus.cc:482] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:55.160506 14134 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.161295 14134 raft_consensus.cc:504] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.161624 14134 leader_election.cc:288] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:55.161939 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:55.161939 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:55.162055 13167 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.162055 12774 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.162943 12774 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:55.162943 13167 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:55.163224 12972 leader_election.cc:302] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.163393 14134 raft_consensus.cc:2781] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:55.163473 14134 raft_consensus.cc:686] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:55.163610 14134 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.165331 12625 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
W20240503 02:00:55.282163 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:55.282456 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:55.285589 14292 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:00:55.372473 12774 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.372516 13038 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.372910 14260 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.373231 14260 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.373703 12774 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.374066 14134 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.376519 13167 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.376703 12907 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.377002 14134 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.377135 14288 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.389653 13169 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.390177 14288 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.592 ms
time per row: 0.0577959 ms
I20240503 02:00:55.441967 12625 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:55.447016 12754 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3bdd27c750447a87691c0f57a70ee1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.447328 12754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3bdd27c750447a87691c0f57a70ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.447728 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 04582c3f86d744ccbc96eec50a10994f (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.447870 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3bdd27c750447a87691c0f57a70ee1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.447993 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.447959 13016 tablet_service.cc:1467] Processing CreateTablet for tablet fdbd697a90534ff3b7a5e17780207d88 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.448109 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3bdd27c750447a87691c0f57a70ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.448478 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbd697a90534ff3b7a5e17780207d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.449620 14277 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:55.450269 14275 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:55.450531 14277 tablet_bootstrap.cc:654] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.450959 14275 tablet_bootstrap.cc:654] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.451849 14277 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:55.451799 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3bdd27c750447a87691c0f57a70ee1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.451923 14277 ts_tablet_manager.cc:1367] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:55.451864 12885 tablet_service.cc:1467] Processing CreateTablet for tablet 04582c3f86d744ccbc96eec50a10994f (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.452059 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3bdd27c750447a87691c0f57a70ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.452180 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.452309 14277 raft_consensus.cc:348] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.452345 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 04582c3f86d744ccbc96eec50a10994f (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.452411 14277 raft_consensus.cc:374] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.452450 14277 raft_consensus.cc:729] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.452593 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.452564 14277 consensus_queue.cc:260] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.452812 14277 ts_tablet_manager.cc:1398] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.452960 14277 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:55.453691 14277 tablet_bootstrap.cc:654] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.454826 14337 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:55.454828 13149 tablet_service.cc:1467] Processing CreateTablet for tablet fdbd697a90534ff3b7a5e17780207d88 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.455075 14277 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:55.455111 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbd697a90534ff3b7a5e17780207d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.455142 14277 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:55.455510 14277 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.455590 14277 raft_consensus.cc:374] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.455614 14277 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.455672 14337 tablet_bootstrap.cc:654] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.455747 14277 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.456063 14277 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:55.456672 14275 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:55.456748 14275 ts_tablet_manager.cc:1367] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:55.457139 14275 raft_consensus.cc:348] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.457252 14275 raft_consensus.cc:374] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.457283 14275 raft_consensus.cc:729] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.457393 14275 consensus_queue.cc:260] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.457669 14275 ts_tablet_manager.cc:1398] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:55.457902 14275 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:55.457957 12756 tablet_service.cc:1467] Processing CreateTablet for tablet fdbd697a90534ff3b7a5e17780207d88 (DEFAULT_TABLE table=rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.458178 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbd697a90534ff3b7a5e17780207d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.459075 14275 tablet_bootstrap.cc:654] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.460145 14335 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:55.460177 14277 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:55.461056 14335 tablet_bootstrap.cc:654] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.461154 14277 tablet_bootstrap.cc:654] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.462055 14337 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:55.462055 14275 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:55.462064 14277 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:55.462131 14277 ts_tablet_manager.cc:1367] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:55.462141 14337 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:55.462150 14275 ts_tablet_manager.cc:1367] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:55.462495 14337 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.462528 14277 raft_consensus.cc:348] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.462528 14275 raft_consensus.cc:348] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.462592 14337 raft_consensus.cc:374] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.462616 14275 raft_consensus.cc:374] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.462616 14277 raft_consensus.cc:374] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.462628 14337 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.462648 14275 raft_consensus.cc:729] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.462666 14277 raft_consensus.cc:729] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.462764 14277 consensus_queue.cc:260] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.462762 14275 consensus_queue.cc:260] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.462774 14337 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.463021 14277 ts_tablet_manager.cc:1398] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:55.463017 14275 ts_tablet_manager.cc:1398] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.463145 14337 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.463311 14337 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:55.463325 14335 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:55.463474 14335 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:55.463987 14335 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.464144 14335 raft_consensus.cc:374] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.464227 14335 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.464418 14335 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.464848 14335 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.465041 14335 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:55.465901 14337 tablet_bootstrap.cc:654] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.466090 14335 tablet_bootstrap.cc:654] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.466987 14337 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:55.467056 14337 ts_tablet_manager.cc:1367] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:55.467378 14337 raft_consensus.cc:348] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.467458 14337 raft_consensus.cc:374] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.467478 14337 raft_consensus.cc:729] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.467594 14337 consensus_queue.cc:260] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.467897 14337 ts_tablet_manager.cc:1398] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.468123 14335 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:55.468202 14335 ts_tablet_manager.cc:1367] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:55.468885 14335 raft_consensus.cc:348] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.468971 14335 raft_consensus.cc:374] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.469007 14335 raft_consensus.cc:729] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.469131 14335 consensus_queue.cc:260] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.469386 14335 ts_tablet_manager.cc:1398] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.515627 14134 raft_consensus.cc:482] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:55.515745 14134 raft_consensus.cc:504] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.516239 14134 leader_election.cc:288] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:55.516585 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbd697a90534ff3b7a5e17780207d88" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:55.516615 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbd697a90534ff3b7a5e17780207d88" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:55.516716 12776 raft_consensus.cc:2454] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:55.516784 13167 raft_consensus.cc:2454] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:55.517012 12972 leader_election.cc:302] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.517171 14134 raft_consensus.cc:2781] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:55.517233 14134 raft_consensus.cc:482] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:55.517261 14134 raft_consensus.cc:3037] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.518018 14134 raft_consensus.cc:504] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.518309 14134 leader_election.cc:288] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:00:55.518630 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbd697a90534ff3b7a5e17780207d88" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:55.518649 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbd697a90534ff3b7a5e17780207d88" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:55.518735 12776 raft_consensus.cc:3037] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.518747 13167 raft_consensus.cc:3037] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.519541 12776 raft_consensus.cc:2454] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:55.519551 13167 raft_consensus.cc:2454] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:55.519850 12972 leader_election.cc:302] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.520020 14134 raft_consensus.cc:2781] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:55.520109 14134 raft_consensus.cc:686] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:55.520242 14134 consensus_queue.cc:237] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:00:55.521977 12629 catalog_manager.cc:5484] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:00:55.553323 14263 raft_consensus.cc:482] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:55.553448 14263 raft_consensus.cc:504] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.553838 14263 leader_election.cc:288] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:55.554186 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04582c3f86d744ccbc96eec50a10994f" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:55.554189 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04582c3f86d744ccbc96eec50a10994f" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:55.554304 12907 raft_consensus.cc:2454] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:55.554323 12776 raft_consensus.cc:2454] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 0.
I20240503 02:00:55.554580 13104 leader_election.cc:302] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:55.554746 14263 raft_consensus.cc:2781] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:55.554800 14263 raft_consensus.cc:482] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:55.554845 14263 raft_consensus.cc:3037] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.555689 14263 raft_consensus.cc:504] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.556015 14263 leader_election.cc:288] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:55.556331 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04582c3f86d744ccbc96eec50a10994f" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:55.556351 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04582c3f86d744ccbc96eec50a10994f" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:55.556425 12776 raft_consensus.cc:3037] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.556457 12907 raft_consensus.cc:3037] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.557153 12776 raft_consensus.cc:2454] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:55.557157 12907 raft_consensus.cc:2454] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 1.
I20240503 02:00:55.557425 13103 leader_election.cc:302] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.557569 14263 raft_consensus.cc:2781] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:55.557659 14263 raft_consensus.cc:686] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:00:55.557801 14263 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.559484 12629 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:00:55.561950 14212 raft_consensus.cc:482] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:55.562043 14212 raft_consensus.cc:504] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.562391 14212 leader_election.cc:288] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:55.562685 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3bdd27c750447a87691c0f57a70ee1" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:55.562755 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3bdd27c750447a87691c0f57a70ee1" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:55.562808 12907 raft_consensus.cc:2454] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:55.562883 13038 raft_consensus.cc:2454] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 0.
I20240503 02:00:55.563112 12711 leader_election.cc:302] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.563288 14212 raft_consensus.cc:2781] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:55.563338 14212 raft_consensus.cc:482] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:55.563380 14212 raft_consensus.cc:3037] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.564194 14212 raft_consensus.cc:504] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.564486 14212 leader_election.cc:288] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:55.564769 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3bdd27c750447a87691c0f57a70ee1" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:55.564808 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3bdd27c750447a87691c0f57a70ee1" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:55.564860 12907 raft_consensus.cc:3037] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.564942 13038 raft_consensus.cc:3037] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:55.565559 12907 raft_consensus.cc:2454] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:55.565675 13038 raft_consensus.cc:2454] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 1.
I20240503 02:00:55.565836 12711 leader_election.cc:302] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:55.566066 14212 raft_consensus.cc:2781] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:55.566220 14212 raft_consensus.cc:686] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:00:55.566345 14212 consensus_queue.cc:237] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.568011 12629 catalog_manager.cc:5484] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
W20240503 02:00:55.681579 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:55.681828 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:55.684837 14349 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 02:00:55.770231 12776 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.770722 13167 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.771426 14134 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.771816 14330 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.778208 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.778839 12774 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.778913 13038 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.779228 14263 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.779366 12907 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:55.779503 14212 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.779968 14263 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:55.780063 14212 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.261 ms
time per row: 0.0591304 ms
I20240503 02:00:55.844077 12629 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44876:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:55.849383 13016 tablet_service.cc:1467] Processing CreateTablet for tablet a105660c12094c71b3a4b80e661d9308 (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.849759 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.849969 12885 tablet_service.cc:1467] Processing CreateTablet for tablet 1c464b6e30bb414bbe70ed030c5f481c (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.850245 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c464b6e30bb414bbe70ed030c5f481c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.850438 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 662782d61c6f472baab4d613b7e6f5cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.850697 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662782d61c6f472baab4d613b7e6f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.851482 14275 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:55.851675 12886 tablet_service.cc:1467] Processing CreateTablet for tablet a105660c12094c71b3a4b80e661d9308 (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.852413 14275 tablet_bootstrap.cc:654] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.852699 13017 tablet_service.cc:1467] Processing CreateTablet for tablet 662782d61c6f472baab4d613b7e6f5cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.852828 14277 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:00:55.852903 13017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662782d61c6f472baab4d613b7e6f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.853246 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.853724 14277 tablet_bootstrap.cc:654] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.851986 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 662782d61c6f472baab4d613b7e6f5cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.854568 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662782d61c6f472baab4d613b7e6f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.855299 14277 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16: No bootstrap required, opened a new log
I20240503 02:00:55.855386 14277 ts_tablet_manager.cc:1367] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:55.855894 14277 raft_consensus.cc:348] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.856004 14277 raft_consensus.cc:374] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.856040 14277 raft_consensus.cc:729] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.856001 13148 tablet_service.cc:1467] Processing CreateTablet for tablet 1c464b6e30bb414bbe70ed030c5f481c (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.852439 14335 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:55.856158 14277 consensus_queue.cc:260] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.856009 13149 tablet_service.cc:1467] Processing CreateTablet for tablet a105660c12094c71b3a4b80e661d9308 (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.856256 13148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c464b6e30bb414bbe70ed030c5f481c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.856482 13016 tablet_service.cc:1467] Processing CreateTablet for tablet 1c464b6e30bb414bbe70ed030c5f481c (DEFAULT_TABLE table=rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:55.856709 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c464b6e30bb414bbe70ed030c5f481c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.857092 14335 tablet_bootstrap.cc:654] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.858188 14335 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:55.858259 14335 ts_tablet_manager.cc:1367] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:00:55.858709 14335 raft_consensus.cc:348] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.858811 14335 raft_consensus.cc:374] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.858843 14335 raft_consensus.cc:729] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.858883 14277 ts_tablet_manager.cc:1398] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:55.858907 14337 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:55.858955 14335 consensus_queue.cc:260] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.859203 14335 ts_tablet_manager.cc:1398] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.859357 14335 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:55.859701 14337 tablet_bootstrap.cc:654] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.860522 14335 tablet_bootstrap.cc:654] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.860750 14275 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:55.860831 14275 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:00:55.860894 14337 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:55.860966 14337 ts_tablet_manager.cc:1367] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:55.861285 14275 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.861377 14275 raft_consensus.cc:374] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.861407 14275 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.861404 14337 raft_consensus.cc:348] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.861507 14337 raft_consensus.cc:374] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.861550 14337 raft_consensus.cc:729] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.861510 14275 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.861670 14337 consensus_queue.cc:260] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.861822 14275 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.861946 14337 ts_tablet_manager.cc:1398] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.862002 14275 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:55.862174 14335 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:55.862241 14335 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:55.862655 14335 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.862754 14335 raft_consensus.cc:374] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.862793 14335 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.862828 14275 tablet_bootstrap.cc:654] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.862910 14335 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.863193 14335 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:55.863350 14335 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:00:55.863848 14275 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:55.863929 14275 ts_tablet_manager.cc:1367] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:55.864228 14335 tablet_bootstrap.cc:654] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.864353 14275 raft_consensus.cc:348] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.864464 14275 raft_consensus.cc:374] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.864501 14275 raft_consensus.cc:729] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.864619 14275 consensus_queue.cc:260] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.864912 14275 ts_tablet_manager.cc:1398] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.865064 14275 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:00:55.865324 13149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:55.865949 14275 tablet_bootstrap.cc:654] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.866925 14275 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: No bootstrap required, opened a new log
I20240503 02:00:55.867000 14275 ts_tablet_manager.cc:1367] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:55.867385 14335 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: No bootstrap required, opened a new log
I20240503 02:00:55.867406 14275 raft_consensus.cc:348] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.867468 14335 ts_tablet_manager.cc:1367] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:55.867508 14275 raft_consensus.cc:374] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.867544 14275 raft_consensus.cc:729] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.867668 14275 consensus_queue.cc:260] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:55.867918 14335 raft_consensus.cc:348] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.868011 14335 raft_consensus.cc:374] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.868028 14275 ts_tablet_manager.cc:1398] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:55.868041 14335 raft_consensus.cc:729] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.868189 14335 consensus_queue.cc:260] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:55.868484 14335 ts_tablet_manager.cc:1398] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:55.868901 14337 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:00:55.870486 14337 tablet_bootstrap.cc:654] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:55.871711 14337 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: No bootstrap required, opened a new log
I20240503 02:00:55.871836 14337 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:55.872299 14337 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.872403 14337 raft_consensus.cc:374] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:55.872447 14337 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:00:55.872560 14337 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:55.872781 14337 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:56.050328 14260 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:56.050458 14260 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:56.050949 14260 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:56.051352 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:00:56.051367 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16" is_pre_election: true
I20240503 02:00:56.051487 13036 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:56.051520 12774 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 0.
I20240503 02:00:56.051817 12843 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:56.051990 14260 raft_consensus.cc:2781] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:56.052047 14260 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:56.052086 14260 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.052891 14260 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:56.053184 14260 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:56.053488 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:00:56.053501 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:00:56.053584 12774 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.053592 13036 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.054288 12774 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:56.054351 13036 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 1.
I20240503 02:00:56.054548 12841 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:00:56.054715 14260 raft_consensus.cc:2781] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:56.054875 14260 raft_consensus.cc:686] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:00:56.055027 14260 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:00:56.056841 12628 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:00:56.064563 14134 raft_consensus.cc:482] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:56.064659 14134 raft_consensus.cc:504] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:56.064992 14134 leader_election.cc:288] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:56.065378 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c464b6e30bb414bbe70ed030c5f481c" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:56.065403 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c464b6e30bb414bbe70ed030c5f481c" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:56.065523 13167 raft_consensus.cc:2454] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:56.065536 12905 raft_consensus.cc:2454] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:56.065847 12973 leader_election.cc:302] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:56.066028 14134 raft_consensus.cc:2781] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:56.066095 14134 raft_consensus.cc:482] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:56.066148 14134 raft_consensus.cc:3037] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.066952 14134 raft_consensus.cc:504] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:56.067263 14134 leader_election.cc:288] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:56.067636 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c464b6e30bb414bbe70ed030c5f481c" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:56.067636 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c464b6e30bb414bbe70ed030c5f481c" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:56.067744 12905 raft_consensus.cc:3037] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.067773 13167 raft_consensus.cc:3037] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.068635 13167 raft_consensus.cc:2454] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:56.068635 12905 raft_consensus.cc:2454] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:56.068938 12973 leader_election.cc:302] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:56.069089 14134 raft_consensus.cc:2781] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:56.069175 14134 raft_consensus.cc:686] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:56.069293 14134 consensus_queue.cc:237] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:00:56.070926 12628 catalog_manager.cc:5484] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:00:56.094923 14134 raft_consensus.cc:482] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:56.095044 14134 raft_consensus.cc:504] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:56.095438 14134 leader_election.cc:288] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:56.095858 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105660c12094c71b3a4b80e661d9308" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:00:56.095880 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105660c12094c71b3a4b80e661d9308" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175" is_pre_election: true
I20240503 02:00:56.096009 13167 raft_consensus.cc:2454] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:56.096010 12905 raft_consensus.cc:2454] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 0.
I20240503 02:00:56.096277 12973 leader_election.cc:302] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:00:56.096427 14134 raft_consensus.cc:2781] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:56.096485 14134 raft_consensus.cc:482] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:56.096535 14134 raft_consensus.cc:3037] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.097292 14134 raft_consensus.cc:504] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:56.097563 14134 leader_election.cc:288] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:56.097927 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105660c12094c71b3a4b80e661d9308" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:00:56.097924 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a105660c12094c71b3a4b80e661d9308" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:00:56.098038 12905 raft_consensus.cc:3037] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.098038 13167 raft_consensus.cc:3037] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:56.098953 13167 raft_consensus.cc:2454] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:56.098953 12905 raft_consensus.cc:2454] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 1.
I20240503 02:00:56.099228 12973 leader_election.cc:302] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5; no voters:
I20240503 02:00:56.099396 14134 raft_consensus.cc:2781] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:56.099493 14134 raft_consensus.cc:686] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:00:56.099635 14134 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:00:56.101229 12628 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:00:56.255604 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:56.255863 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:56.258761 14402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 02:00:56.333849 12905 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.334280 14134 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.335057 13167 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.335892 14134 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.347980 13036 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.348134 12774 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.348517 14398 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.348652 14260 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.352672 13167 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.355508 14134 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.361683 12905 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:56.362167 14134 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:56.380360 14447 mvcc.cc:205] Tried to move back new op lower bound from 7023417984396988416 to 7023417983202705408. Current Snapshot: MvccSnapshot[applied={T|T < 7023417984396988416}]
I20240503 02:00:56.381883 14438 mvcc.cc:205] Tried to move back new op lower bound from 7023417984374493184 to 7023417983385292800. Current Snapshot: MvccSnapshot[applied={T|T < 7023417984374493184}]
INSERT report
rows total: 2000
time total: 116.298 ms
time per row: 0.0581489 ms
I20240503 02:00:56.416240 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:37055
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:37619
--webserver_port=46583
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:56.524569 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:56.524766 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:56.524814 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:56.527812 14458 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:56.527870 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:56.527963 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:00:56.532079 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:37055
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=46583
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.14458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:56.533156 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:56.534273 14458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:56.543015 14463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.544127 14466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.544961 14464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:56.545781 14458 server_base.cc:1034] running on GCE node
I20240503 02:00:56.546191 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:56.546742 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:56.547888 14458 hybrid_clock.cc:648] HybridClock initialized: now 1714701656547868 us; error 43 us; skew 500 ppm
I20240503 02:00:56.555227 14458 webserver.cc:460] Webserver started at http://127.0.99.133:46583/ using document root <none> and password file <none>
I20240503 02:00:56.555850 14458 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:56.555920 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:56.559247 14458 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:56.561097 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.562114 14458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:56.562209 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9e39ab9b64544988b968015fe9fa06e6"
format_stamp: "Formatted at 2024-05-03 02:00:46 on dist-test-slave-fvr5"
I20240503 02:00:56.562600 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:56.592000 14458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:56.592617 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:56.592754 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:56.593248 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:56.594163 14458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:56.594215 14458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.594272 14458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:56.594303 14458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.603175 14458 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37055
I20240503 02:00:56.603212 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37055 every 8 connection(s)
I20240503 02:00:56.604233 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:00:56.612396 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 14458
I20240503 02:00:56.612567 14586 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:56.612788 14586 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:56.612871 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:40697
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:37619
--webserver_port=37021
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:56.613307 14586 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:56.614703 12628 ts_manager.cc:194] Re-registered known tserver with Master: 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:00:56.615242 12628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:57647
W20240503 02:00:56.728148 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:56.728395 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:56.728457 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:56.732604 14589 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:56.732676 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:56.732765 14589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:00:56.737581 14589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:40697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=37021
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.14589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:56.738582 14589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:56.739889 14589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:56.750735 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.751789 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.752552 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:56.753420 14589 server_base.cc:1034] running on GCE node
I20240503 02:00:56.753799 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:56.754339 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:56.755506 14589 hybrid_clock.cc:648] HybridClock initialized: now 1714701656755487 us; error 32 us; skew 500 ppm
I20240503 02:00:56.757458 14589 webserver.cc:460] Webserver started at http://127.0.99.134:37021/ using document root <none> and password file <none>
I20240503 02:00:56.757988 14589 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:56.758062 14589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:56.761301 14589 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:56.763039 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.764052 14589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:56.764158 14589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:56.764672 14589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:56.799484 14589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:56.800158 14589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:56.800328 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:56.800961 14589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:56.801951 14589 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:56.802004 14589 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.802069 14589 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:56.802099 14589 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.811805 14589 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40697
I20240503 02:00:56.812117 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40697 every 8 connection(s)
I20240503 02:00:56.812893 14589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:00:56.819111 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 14589
I20240503 02:00:56.819620 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:37171
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:37619
--webserver_port=44591
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:00:56.833741 14717 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:56.834038 14717 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:56.834738 14717 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:56.835997 12628 ts_manager.cc:194] Re-registered known tserver with Master: d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:00:56.836699 12628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:50027
W20240503 02:00:56.933967 14720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:56.934162 14720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:56.934209 14720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:56.937386 14720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:56.937448 14720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:56.937527 14720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:00:56.941135 14720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:37171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=44591
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.14720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:56.941956 14720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:56.943003 14720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:56.951606 14728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.952402 14725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:56.953505 14726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:56.954547 14720 server_base.cc:1034] running on GCE node
I20240503 02:00:56.954959 14720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:56.955513 14720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:56.956673 14720 hybrid_clock.cc:648] HybridClock initialized: now 1714701656956634 us; error 41 us; skew 500 ppm
I20240503 02:00:56.958528 14720 webserver.cc:460] Webserver started at http://127.0.99.135:44591/ using document root <none> and password file <none>
I20240503 02:00:56.959072 14720 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:56.959139 14720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:56.962666 14720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:56.964485 14734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.965556 14720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:56.965667 14720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "754fc04ea9914073a56c0cf164f63240"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:56.966091 14720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:56.985069 14720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:56.985605 14720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:56.985760 14720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:56.986285 14720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:56.987125 14720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:56.987174 14720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.987232 14720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:56.987262 14720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:56.995800 14720 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:37171
I20240503 02:00:56.995831 14847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:37171 every 8 connection(s)
I20240503 02:00:56.996784 14720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:00:57.004706 14848 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:57.004897 14848 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:57.005280 14848 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:57.006227 12628 ts_manager.cc:194] Re-registered known tserver with Master: 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:00:57.006703 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 14720
I20240503 02:00:57.006789 12628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:45319
I20240503 02:00:57.007141 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41447
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:37619
--webserver_port=45657
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:00:57.115478 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.115803 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:00:57.115886 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.119024 14851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:00:57.119081 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:00:57.119145 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 02:00:57.122921 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41447
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=45657
--tserver_master_addrs=127.0.99.190:37619
--never_fsync=true
--heap_profile_path=/tmp/kudu.14851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:00:57.123776 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:00:57.124838 14851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:00:57.133406 14856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:57.134029 14857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:00:57.134789 14859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:00:57.135861 14851 server_base.cc:1034] running on GCE node
I20240503 02:00:57.136730 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:00:57.137336 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:00:57.138513 14851 hybrid_clock.cc:648] HybridClock initialized: now 1714701657138485 us; error 44 us; skew 500 ppm
I20240503 02:00:57.140651 14851 webserver.cc:460] Webserver started at http://127.0.99.136:45657/ using document root <none> and password file <none>
I20240503 02:00:57.141191 14851 fs_manager.cc:359] Metadata directory not provided
I20240503 02:00:57.141274 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:00:57.144608 14851 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:57.146384 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:57.147405 14851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:57.147536 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "558837c679f246dcb3241d66a37d70f1"
format_stamp: "Formatted at 2024-05-03 02:00:47 on dist-test-slave-fvr5"
I20240503 02:00:57.148029 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:00:57.161978 14851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:00:57.162523 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:00:57.162674 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:00:57.163162 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:00:57.164027 14851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:00:57.164073 14851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:57.164119 14851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:00:57.164149 14851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:00:57.173691 14851 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41447
I20240503 02:00:57.173763 14979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41447 every 8 connection(s)
I20240503 02:00:57.174680 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:00:57.182659 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 14851
I20240503 02:00:57.183818 14980 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37619
I20240503 02:00:57.184037 14980 heartbeater.cc:464] Registering TS with master...
I20240503 02:00:57.184535 14980 heartbeater.cc:511] Master 127.0.99.190:37619 requested a full tablet report, sending...
I20240503 02:00:57.185541 12628 ts_manager.cc:194] Re-registered known tserver with Master: 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:00:57.186198 12628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:40327
I20240503 02:00:57.540138 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.547482 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.563865 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.564143 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:57.569968 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:57.577008 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:57.582720 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:57.594231 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:57.599709 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.609699 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.616255 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:57.616576 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.617295 14586 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:57.633538 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:57.640235 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.644389 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:57.660327 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.714835 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:57.720778 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.734786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:57.738641 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.749122 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:57.766402 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.770324 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:57.786485 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.840350 14717 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:57.846002 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.849714 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:57.859619 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:57.860958 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.866361 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:57.867049 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.872778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:57.873714 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.877543 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.880275 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:57.890105 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:57.895846 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:57.899910 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.922261 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:57.934166 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:57.945608 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.008798 14848 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:58.098985 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.100214 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.115475 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.128713 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.129741 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.144415 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.155545 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.156229 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.160964 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.178572 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.189751 14980 heartbeater.cc:503] Master 127.0.99.190:37619 was elected leader, sending a full tablet report...
I20240503 02:00:58.198686 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.201304 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.201601 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.207901 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.216238 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.219513 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.219899 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.228514 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.239161 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.243402 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.268368 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.277783 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.291958 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.297564 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.300282 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.300532 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.313119 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.316454 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.334601 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.335284 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.347064 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.390816 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.436589 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.445282 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.446419 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.446460 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.448642 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.456346 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.457404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.468189 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.586699 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.592653 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.605643 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.629979 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.647996 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.656862 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.665839 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.670045 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.677776 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.684525 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.688879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.696115 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.705951 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.720544 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.723117 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.732256 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.733860 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.736600 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.737260 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.745261 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.765690 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.767292 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.778760 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.780171 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.781497 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.787868 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.794497 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.827641 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.833032 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.835796 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.846812 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.860595 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.929446 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:58.930150 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.938511 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:58.942657 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.949029 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:58.957033 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:58.976491 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:58.979283 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.015300 13169 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.018359 12774 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.019052 13036 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.019701 14263 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.021392 14263 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.026849 12776 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.027926 15772 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.029882 12630 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.030994 13036 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
W20240503 02:00:59.034533 13103 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 063a7f2ed28a455697b000f52a4343a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.036309 12905 consensus_queue.cc:237] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.036623 13036 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.038029 12776 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.038568 12776 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.038620 12905 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.039041 15274 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.039163 15196 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.039211 14866 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.039868 13036 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.041163 15665 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.043061 15196 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.043447 13038 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.043808 15274 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.045251 12774 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.046439 12625 catalog_manager.cc:5484] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.047236 12776 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.051662 12905 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:00:59.053372 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86b733f38fd4f08bd40865b99e2e7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.054538 12628 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.057147 13169 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.059172 12776 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.059574 14263 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.061456 12905 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.061826 14263 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:59.063223 12842 consensus_peers.cc:485] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79c5f35eb8b546de8015c5235913efce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.064049 14263 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.066022 12776 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
W20240503 02:00:59.066766 13105 consensus_peers.cc:485] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4b734308d3b42958e3a2830a83d44e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.067394 12625 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.068701 12905 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.090090 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.091562 13167 consensus_queue.cc:237] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.091663 12905 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.091989 13169 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:59.092072 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a105660c12094c71b3a4b80e661d9308. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.092368 15274 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.092551 15800 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.093993 12907 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.094026 13038 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.094406 14263 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.095602 14263 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.096107 13169 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.097681 12776 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:59.097817 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:59.098176 13103 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 063a7f2ed28a455697b000f52a4343a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.098220 15772 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:00:59.098819 13105 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 063a7f2ed28a455697b000f52a4343a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.099347 15772 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:59.101133 15772 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
W20240503 02:00:59.101490 13104 consensus_peers.cc:485] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6cc67c73560449e81303a948df8f9df. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.102636 15771 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.103940 12905 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.104406 12625 catalog_manager.cc:5484] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.104493 12776 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.107515 13038 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.107563 13036 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.107633 12632 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.112442 15800 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.113204 13167 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.117063 12628 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.117476 12905 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.144028 13038 consensus_queue.cc:237] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
W20240503 02:00:59.146250 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c464b6e30bb414bbe70ed030c5f481c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.146922 13167 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.147270 12905 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.147666 15885 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.148051 15885 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.148877 12907 consensus_queue.cc:237] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.150848 13038 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:00:59.151064 12842 consensus_peers.cc:485] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79c5f35eb8b546de8015c5235913efce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.151294 15665 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:59.152005 12776 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:00:59.153105 15965 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:59.156051 15665 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
W20240503 02:00:59.158098 12841 consensus_peers.cc:485] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79c5f35eb8b546de8015c5235913efce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.158453 12776 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.159307 15885 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.159529 13038 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.161298 13167 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.163182 12628 catalog_manager.cc:5484] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.163697 12905 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.165263 12625 catalog_manager.cc:5484] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.200496 13167 consensus_queue.cc:237] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.202098 13038 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:59.202253 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0633596226954c439d2f671c25eebfe6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.202838 12776 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.203080 15883 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.203677 15883 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.207273 15883 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.208631 13038 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.213677 12776 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.215813 12633 catalog_manager.cc:5484] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.223618 12905 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.225349 13167 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:59.226791 12842 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0613ddd56ed346b3b6dfa9e7c3a4335e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.226904 15965 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.228184 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.229156 12776 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.230998 15965 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.232553 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:59.232692 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.234795 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.242259 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:59.242321 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:59.244217 15965 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.249277 12625 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.258543 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.258589 13167 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.263136 12776 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.265175 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:59.361982 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:59.371345 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:59.371484 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.376919 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.377275 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.395359 13038 consensus_queue.cc:237] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.397294 12774 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.397974 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:59.399685 13167 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.401819 13036 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.402154 12905 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.402541 15882 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:59.403206 13103 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 141097686fe24c8f9956dcc1ad1d253c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.404722 15882 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.404929 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.405201 15274 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.405422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:59.409235 13167 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.409719 15274 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:59.410166 12972 consensus_peers.cc:485] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdbd697a90534ff3b7a5e17780207d88. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.421190 15882 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.423274 12905 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.425750 12632 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.426244 13036 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.436161 14866 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.437927 13167 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.438555 12630 catalog_manager.cc:5484] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.439952 12774 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.471798 16170 ts_tablet_manager.cc:902] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:59.472836 16170 tablet_copy_client.cc:308] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:00:59.480476 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet d86b733f38fd4f08bd40865b99e2e7b2 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:00:59.480636 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet d86b733f38fd4f08bd40865b99e2e7b2 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-d86b733f38fd4f08bd40865b99e2e7b2
I20240503 02:00:59.482221 13055 tablet_copy_source_session.cc:205] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.486653 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b733f38fd4f08bd40865b99e2e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.493952 16170 tablet_copy_client.cc:783] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.494268 16170 tablet_copy_client.cc:647] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.496049 16170 tablet_copy_client.cc:523] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.500064 16170 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:00:59.509141 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:59.512341 12905 consensus_queue.cc:237] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.513973 13036 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.514824 15665 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:59.515321 12842 consensus_peers.cc:485] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b39fa1cb3863414287a47a0832c8b55d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.515580 13036 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.516461 12774 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.517024 15196 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.517139 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:59.519094 12905 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.519536 12776 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.520342 15885 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.520423 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.520893 15885 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.527595 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:00:59.529975 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34e88e69beb64036bb676b771dabd2a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.535027 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:59.535602 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.535894 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:00:59.550515 16170 log.cc:824] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:59.550895 15885 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.551126 16210 ts_tablet_manager.cc:902] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:59.551944 16210 tablet_copy_client.cc:308] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:00:59.553366 12905 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.553650 12776 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.556078 12629 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.557338 15196 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.559335 13038 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.561579 12774 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.564107 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.565176 16219 ts_tablet_manager.cc:902] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:59.565742 16219 tablet_copy_client.cc:308] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:00:59.569790 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 063a7f2ed28a455697b000f52a4343a3 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:57047)
I20240503 02:00:59.569954 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 063a7f2ed28a455697b000f52a4343a3 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:57047: session id = 754fc04ea9914073a56c0cf164f63240-063a7f2ed28a455697b000f52a4343a3
I20240503 02:00:59.571287 13189 tablet_copy_source_session.cc:205] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.573474 12631 catalog_manager.cc:5484] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.577629 16210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.579000 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:00:59.588882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:59.590713 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 79c5f35eb8b546de8015c5235913efce from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:57887)
I20240503 02:00:59.590842 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 79c5f35eb8b546de8015c5235913efce from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:57887: session id = 754fc04ea9914073a56c0cf164f63240-79c5f35eb8b546de8015c5235913efce
I20240503 02:00:59.592265 12927 tablet_copy_source_session.cc:205] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.590507 16243 ts_tablet_manager.cc:902] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:59.594003 16243 tablet_copy_client.cc:308] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:00:59.600172 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79c5f35eb8b546de8015c5235913efce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.600394 16210 tablet_copy_client.cc:783] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.600672 16210 tablet_copy_client.cc:647] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.601771 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.603861 16219 tablet_copy_client.cc:783] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.604128 16210 tablet_copy_client.cc:523] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.606030 16210 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:00:59.615010 16219 tablet_copy_client.cc:647] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.622154 16219 tablet_copy_client.cc:523] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.623705 16219 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:00:59.629706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:00:59.630724 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 063a7f2ed28a455697b000f52a4343a3 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:53947)
I20240503 02:00:59.630846 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 063a7f2ed28a455697b000f52a4343a3 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:53947: session id = 9e39ab9b64544988b968015fe9fa06e6-063a7f2ed28a455697b000f52a4343a3
I20240503 02:00:59.633436 13189 tablet_copy_source_session.cc:205] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.634016 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.635468 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.636339 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:00:59.638922 16243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.656489 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:00:59.658567 16210 log.cc:824] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:59.661751 16255 ts_tablet_manager.cc:902] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:59.662992 16255 tablet_copy_client.cc:308] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:00:59.683924 16256 ts_tablet_manager.cc:902] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:59.684563 16256 tablet_copy_client.cc:308] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:00:59.685111 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet a105660c12094c71b3a4b80e661d9308 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:00:59.685209 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet a105660c12094c71b3a4b80e661d9308 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-a105660c12094c71b3a4b80e661d9308
I20240503 02:00:59.686204 13055 tablet_copy_source_session.cc:205] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.687975 16256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.688110 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 79c5f35eb8b546de8015c5235913efce from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:49001)
I20240503 02:00:59.688220 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 79c5f35eb8b546de8015c5235913efce from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:49001: session id = d23425fa2b5848c8b1a2b737d75e69eb-79c5f35eb8b546de8015c5235913efce
I20240503 02:00:59.689126 12927 tablet_copy_source_session.cc:205] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.690688 16257 ts_tablet_manager.cc:902] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:59.691347 16257 tablet_copy_client.cc:308] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:00:59.691931 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet b4b734308d3b42958e3a2830a83d44e2 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:53947)
I20240503 02:00:59.692047 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet b4b734308d3b42958e3a2830a83d44e2 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:53947: session id = 9e39ab9b64544988b968015fe9fa06e6-b4b734308d3b42958e3a2830a83d44e2
I20240503 02:00:59.692948 13189 tablet_copy_source_session.cc:205] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.693341 16255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79c5f35eb8b546de8015c5235913efce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.694008 16257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b734308d3b42958e3a2830a83d44e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.694844 16243 tablet_copy_client.cc:783] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.695075 16243 tablet_copy_client.cc:647] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.697643 16257 tablet_copy_client.cc:783] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.697876 16257 tablet_copy_client.cc:647] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.699918 16256 tablet_copy_client.cc:783] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.700135 16256 tablet_copy_client.cc:647] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.700927 16255 tablet_copy_client.cc:783] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.701121 16255 tablet_copy_client.cc:647] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.701259 16257 tablet_copy_client.cc:523] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.702723 16255 tablet_copy_client.cc:523] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.703370 16257 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:00:59.703694 16210 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.704115 16210 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:00:59.704430 16210 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.099s user 0.030s sys 0.005s
I20240503 02:00:59.706064 16255 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:00:59.707919 16210 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.708192 16210 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.708355 16243 tablet_copy_client.cc:523] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.708782 16282 ts_tablet_manager.cc:902] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:59.708758 16210 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.709606 16210 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:59.709785 16243 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:00:59.710191 16282 tablet_copy_client.cc:308] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:00:59.710315 16257 log.cc:824] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:59.711334 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-063a7f2ed28a455697b000f52a4343a3 received from {username='slave'} at 127.0.99.135:57047
I20240503 02:00:59.711429 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-063a7f2ed28a455697b000f52a4343a3 on tablet 063a7f2ed28a455697b000f52a4343a3 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.712329 16256 tablet_copy_client.cc:523] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.713452 16256 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:00:59.715160 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet e6cc67c73560449e81303a948df8f9df from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:41125)
I20240503 02:00:59.715267 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet e6cc67c73560449e81303a948df8f9df from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:41125: session id = d23425fa2b5848c8b1a2b737d75e69eb-e6cc67c73560449e81303a948df8f9df
I20240503 02:00:59.716281 13189 tablet_copy_source_session.cc:205] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.718060 16282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cc67c73560449e81303a948df8f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.720682 16282 tablet_copy_client.cc:783] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.720882 16282 tablet_copy_client.cc:647] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.726909 13038 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.728996 13167 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.729068 12907 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:59.729450 12972 consensus_peers.cc:485] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86e314466c01488bb661c596c405db51. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.729640 15274 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.730798 15274 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.732092 16288 ts_tablet_manager.cc:902] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:59.732679 16288 tablet_copy_client.cc:308] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:00:59.733235 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 1c464b6e30bb414bbe70ed030c5f481c from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:00:59.733325 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 1c464b6e30bb414bbe70ed030c5f481c from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-1c464b6e30bb414bbe70ed030c5f481c
I20240503 02:00:59.734270 13055 tablet_copy_source_session.cc:205] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.735335 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c464b6e30bb414bbe70ed030c5f481c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.741231 16282 tablet_copy_client.cc:523] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.742573 16282 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:00:59.743340 16255 log.cc:824] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Log is configured to *not* fsync() on all Append() calls
I20240503 02:00:59.748719 16257 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.749050 16257 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:00:59.749344 16257 ts_tablet_manager.cc:1367] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.046s user 0.036s sys 0.000s
I20240503 02:00:59.750331 16170 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.753459 16257 raft_consensus.cc:348] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.753700 16257 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.754191 16257 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.751125 16288 tablet_copy_client.cc:783] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.754521 16288 tablet_copy_client.cc:647] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.755034 16257 ts_tablet_manager.cc:1398] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:00:59.755656 16288 tablet_copy_client.cc:523] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.757236 16288 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:00:59.757993 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-b4b734308d3b42958e3a2830a83d44e2 received from {username='slave'} at 127.0.99.133:53947
I20240503 02:00:59.758076 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-b4b734308d3b42958e3a2830a83d44e2 on tablet b4b734308d3b42958e3a2830a83d44e2 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:00:59.771167 16257 ts_tablet_manager.cc:902] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:00:59.771579 16257 tablet_copy_client.cc:308] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:00:59.772053 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 0633596226954c439d2f671c25eebfe6 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:53947)
I20240503 02:00:59.772176 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 0633596226954c439d2f671c25eebfe6 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:53947: session id = 9e39ab9b64544988b968015fe9fa06e6-0633596226954c439d2f671c25eebfe6
I20240503 02:00:59.773093 13189 tablet_copy_source_session.cc:205] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.773972 16257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0633596226954c439d2f671c25eebfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.776361 16257 tablet_copy_client.cc:783] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.776522 16257 tablet_copy_client.cc:647] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.783862 16219 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.784191 16219 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:00:59.784368 16219 ts_tablet_manager.cc:1367] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.161s user 0.026s sys 0.008s
I20240503 02:00:59.784965 16219 raft_consensus.cc:348] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.785095 16219 raft_consensus.cc:729] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.785225 16219 consensus_queue.cc:260] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.785513 16219 ts_tablet_manager.cc:1398] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:59.786226 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-79c5f35eb8b546de8015c5235913efce received from {username='slave'} at 127.0.99.135:57887
I20240503 02:00:59.786326 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-79c5f35eb8b546de8015c5235913efce on tablet 79c5f35eb8b546de8015c5235913efce with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.793869 13038 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.796140 16170 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:00:59.796214 12905 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.796576 13167 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.796602 16170 ts_tablet_manager.cc:1367] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.299s user 0.040s sys 0.000s
I20240503 02:00:59.797127 14866 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.797442 14866 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.799384 16170 raft_consensus.cc:348] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
W20240503 02:00:59.800827 12973 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 613074d3c4514bd5b64a06c59af09a56. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.801249 16170 raft_consensus.cc:729] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.809581 16257 tablet_copy_client.cc:523] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.810433 16282 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.810834 16282 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:00:59.811094 16257 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:00:59.811226 16282 ts_tablet_manager.cc:1367] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.005s
I20240503 02:00:59.813781 16282 raft_consensus.cc:348] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.814054 16282 raft_consensus.cc:729] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.814534 16282 consensus_queue.cc:260] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.815414 16255 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.815522 16282 ts_tablet_manager.cc:1398] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:59.816164 16255 tablet_bootstrap.cc:492] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:00:59.816368 16255 ts_tablet_manager.cc:1367] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.110s user 0.032s sys 0.001s
I20240503 02:00:59.817056 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-e6cc67c73560449e81303a948df8f9df received from {username='slave'} at 127.0.99.134:41125
I20240503 02:00:59.817137 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-e6cc67c73560449e81303a948df8f9df on tablet e6cc67c73560449e81303a948df8f9df with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.817459 16255 raft_consensus.cc:348] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.817574 16255 raft_consensus.cc:729] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.817719 16255 consensus_queue.cc:260] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.818576 16255 ts_tablet_manager.cc:1398] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:59.819059 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-79c5f35eb8b546de8015c5235913efce received from {username='slave'} at 127.0.99.134:49001
I20240503 02:00:59.819145 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-79c5f35eb8b546de8015c5235913efce on tablet 79c5f35eb8b546de8015c5235913efce with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.819576 14866 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.820169 13167 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.824522 12625 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.826905 12905 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.829434 15800 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.831636 12907 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.831974 12625 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.833168 13167 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.835716 16170 consensus_queue.cc:260] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.836791 16170 ts_tablet_manager.cc:1398] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.040s user 0.000s sys 0.003s
I20240503 02:00:59.837949 16255 ts_tablet_manager.cc:902] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:59.838378 16255 tablet_copy_client.cc:308] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:00:59.838791 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:49001)
I20240503 02:00:59.838876 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:49001: session id = d23425fa2b5848c8b1a2b737d75e69eb-0613ddd56ed346b3b6dfa9e7c3a4335e
I20240503 02:00:59.839725 12927 tablet_copy_source_session.cc:205] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.840750 16255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.840866 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-d86b733f38fd4f08bd40865b99e2e7b2 received from {username='slave'} at 127.0.99.136:51355
I20240503 02:00:59.840941 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-d86b733f38fd4f08bd40865b99e2e7b2 on tablet d86b733f38fd4f08bd40865b99e2e7b2 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:00:59.843904 16255 tablet_copy_client.cc:783] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.844089 16255 tablet_copy_client.cc:647] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.849766 16243 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.850107 16243 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:00:59.850302 16243 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.141s user 0.029s sys 0.000s
I20240503 02:00:59.850886 16243 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.851050 16243 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.851238 16243 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.852032 16243 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:59.852524 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-063a7f2ed28a455697b000f52a4343a3 received from {username='slave'} at 127.0.99.133:53947
I20240503 02:00:59.852602 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-063a7f2ed28a455697b000f52a4343a3 on tablet 063a7f2ed28a455697b000f52a4343a3 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:00:59.857406 16255 tablet_copy_client.cc:523] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.858819 16255 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:00:59.876724 16288 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.877020 16288 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:00:59.877234 16288 ts_tablet_manager.cc:1367] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.120s user 0.026s sys 0.000s
I20240503 02:00:59.877833 16288 raft_consensus.cc:348] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.878556 16288 raft_consensus.cc:729] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.878813 16288 consensus_queue.cc:260] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.879513 16288 ts_tablet_manager.cc:1398] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:59.880533 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-1c464b6e30bb414bbe70ed030c5f481c received from {username='slave'} at 127.0.99.136:51355
I20240503 02:00:59.880620 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-1c464b6e30bb414bbe70ed030c5f481c on tablet 1c464b6e30bb414bbe70ed030c5f481c with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:00:59.886009 16256 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.886335 16256 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:00:59.886547 16256 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.173s user 0.028s sys 0.000s
I20240503 02:00:59.887025 16256 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.887164 16256 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.887356 16256 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.887660 16256 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:59.888126 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-a105660c12094c71b3a4b80e661d9308 received from {username='slave'} at 127.0.99.136:51355
I20240503 02:00:59.888206 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-a105660c12094c71b3a4b80e661d9308 on tablet a105660c12094c71b3a4b80e661d9308 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:00:59.891799 14934 raft_consensus.cc:1203] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:59.894205 16257 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.894479 16257 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:00:59.894657 16257 ts_tablet_manager.cc:1367] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.084s user 0.019s sys 0.004s
I20240503 02:00:59.895057 16257 raft_consensus.cc:348] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.895195 16257 raft_consensus.cc:729] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.895354 16257 consensus_queue.cc:260] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.895732 16257 ts_tablet_manager.cc:1398] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:59.896217 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0633596226954c439d2f671c25eebfe6 received from {username='slave'} at 127.0.99.133:53947
I20240503 02:00:59.896304 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0633596226954c439d2f671c25eebfe6 on tablet 0633596226954c439d2f671c25eebfe6 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:00:59.899361 12907 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.901021 12774 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.901177 13167 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.901445 15665 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:59.901619 12842 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 490776bd408a48a9bcbbc7a501baa8ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.901684 16219 ts_tablet_manager.cc:902] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:00:59.901816 15965 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.902127 16219 tablet_copy_client.cc:308] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:00:59.905014 13038 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.907155 13169 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.907547 15274 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.908012 12776 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:59.909292 12972 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1f9c8efd2040d6842c6b9f7c6173aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.909454 15274 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.909667 16255 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.909953 16255 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:00:59.910198 16255 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.004s
I20240503 02:00:59.910729 16255 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.910907 16255 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.911109 16255 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:00:59.911479 16255 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:59.911924 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-0613ddd56ed346b3b6dfa9e7c3a4335e received from {username='slave'} at 127.0.99.134:49001
I20240503 02:00:59.912001 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-0613ddd56ed346b3b6dfa9e7c3a4335e on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.912277 15274 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.912505 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet fdbd697a90534ff3b7a5e17780207d88 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40691)
I20240503 02:00:59.912609 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet fdbd697a90534ff3b7a5e17780207d88 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40691: session id = 754fc04ea9914073a56c0cf164f63240-fdbd697a90534ff3b7a5e17780207d88
I20240503 02:00:59.913645 13055 tablet_copy_source_session.cc:205] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.914624 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbd697a90534ff3b7a5e17780207d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.915089 12625 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.915203 12774 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.917357 13167 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:00:59.917908 15965 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.920118 13167 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.920653 12625 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.924774 12774 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:00:59.929673 16219 tablet_copy_client.cc:783] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.929858 16219 tablet_copy_client.cc:647] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.929919 14801 raft_consensus.cc:1203] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:00:59.936834 12774 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.937392 12776 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.939059 13167 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.939980 15464 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.941722 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.942112 15464 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.942746 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.943161 13167 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.943913 16219 tablet_copy_client.cc:523] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.944833 15464 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.945216 15464 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.945624 16219 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
W20240503 02:00:59.953083 12710 consensus_peers.cc:485] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007feed148fb44e9a6fe9d449b6d8f9d. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:59.955426 12712 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5767bdfcc6249578f7c3ada3c08da6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:59.957886 13167 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.959734 16453 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.960911 12630 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.962545 15464 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.965561 12907 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.965847 12630 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.970243 13037 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:00:59.973264 16255 ts_tablet_manager.cc:902] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:00:59.973729 16255 tablet_copy_client.cc:308] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:00:59.974157 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet b39fa1cb3863414287a47a0832c8b55d from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:49001)
I20240503 02:00:59.974263 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet b39fa1cb3863414287a47a0832c8b55d from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:49001: session id = d23425fa2b5848c8b1a2b737d75e69eb-b39fa1cb3863414287a47a0832c8b55d
I20240503 02:00:59.975260 12927 tablet_copy_source_session.cc:205] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:59.977384 16255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39fa1cb3863414287a47a0832c8b55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:59.979517 16255 tablet_copy_client.cc:783] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:59.979682 16255 tablet_copy_client.cc:647] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:59.980675 13169 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:00:59.983501 16255 tablet_copy_client.cc:523] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:59.984671 16255 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:00:59.992278 12907 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:00:59.994156 16219 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:59.994486 16219 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:00:59.994534 13038 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.994875 12774 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:59.994915 15965 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.994660 16219 ts_tablet_manager.cc:1367] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.049s user 0.030s sys 0.000s
I20240503 02:00:59.995231 15965 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:59.995458 16219 raft_consensus.cc:348] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.995572 16219 raft_consensus.cc:729] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:00:59.996347 16219 consensus_queue.cc:260] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:00:59.996682 16219 ts_tablet_manager.cc:1398] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:59.997743 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-fdbd697a90534ff3b7a5e17780207d88 received from {username='slave'} at 127.0.99.135:40691
I20240503 02:00:59.997818 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-fdbd697a90534ff3b7a5e17780207d88 on tablet fdbd697a90534ff3b7a5e17780207d88 with peer 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.003285 12843 consensus_peers.cc:485] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.003963 15965 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.005791 13038 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.005908 12774 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.008095 12630 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index -1 to 4, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.017493 16219 ts_tablet_manager.cc:902] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:00.017920 16219 tablet_copy_client.cc:308] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:00.018426 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 141097686fe24c8f9956dcc1ad1d253c from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:57047)
I20240503 02:01:00.018527 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 141097686fe24c8f9956dcc1ad1d253c from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:57047: session id = 754fc04ea9914073a56c0cf164f63240-141097686fe24c8f9956dcc1ad1d253c
I20240503 02:01:00.019506 13189 tablet_copy_source_session.cc:205] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.020543 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.021979 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:00.022933 16219 tablet_copy_client.cc:783] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.023078 16219 tablet_copy_client.cc:647] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.025422 16219 tablet_copy_client.cc:523] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.026556 16219 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:00.038733 12774 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.041041 12907 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.041782 13169 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.045789 16449 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.046164 16449 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.050293 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:01:00.052477 12711 consensus_peers.cc:485] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13a95f88ac1c459ba01be0743297f696. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.052604 14671 raft_consensus.cc:1203] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:00.055632 14801 raft_consensus.cc:1203] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:00.057251 16450 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.059078 16257 ts_tablet_manager.cc:902] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:00.059523 16257 tablet_copy_client.cc:308] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:00.060947 12633 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.061147 13169 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.062893 12907 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.067000 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:00.069191 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 34e88e69beb64036bb676b771dabd2a4 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:54909)
I20240503 02:01:00.069293 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 34e88e69beb64036bb676b771dabd2a4 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:54909: session id = 9e39ab9b64544988b968015fe9fa06e6-34e88e69beb64036bb676b771dabd2a4
I20240503 02:01:00.070216 13055 tablet_copy_source_session.cc:205] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.073786 16255 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.076478 16255 tablet_bootstrap.cc:492] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:00.076714 16255 ts_tablet_manager.cc:1367] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.092s user 0.029s sys 0.000s
I20240503 02:01:00.077204 16257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e88e69beb64036bb676b771dabd2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.079238 16257 tablet_copy_client.cc:783] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.079378 16257 tablet_copy_client.cc:647] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.080531 16255 raft_consensus.cc:348] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.080749 16255 raft_consensus.cc:729] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.085012 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:00.091799 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.091811 16257 tablet_copy_client.cc:523] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.093271 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:00.093290 16257 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:00.095063 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.098335 12774 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
W20240503 02:01:00.100661 12712 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5767bdfcc6249578f7c3ada3c08da6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.101022 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:01:00.101526 12710 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5767bdfcc6249578f7c3ada3c08da6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.101857 16449 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:00.102769 13169 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:00.103269 16449 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:00.105787 16449 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.108641 12630 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.111917 13169 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.114305 13038 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.115226 14539 raft_consensus.cc:1203] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:00.122212 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.125988 16255 consensus_queue.cc:260] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.127066 16255 ts_tablet_manager.cc:1398] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.050s user 0.001s sys 0.000s
I20240503 02:01:00.132139 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-b39fa1cb3863414287a47a0832c8b55d received from {username='slave'} at 127.0.99.134:49001
I20240503 02:01:00.132231 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-b39fa1cb3863414287a47a0832c8b55d on tablet b39fa1cb3863414287a47a0832c8b55d with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:00.163161 13038 consensus_queue.cc:237] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.164687 14539 raft_consensus.cc:1203] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.167825 16257 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.168186 16257 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
W20240503 02:01:00.168361 12972 consensus_peers.cc:485] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8108933516894d4190ef9a0cf6063be8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.168366 16257 ts_tablet_manager.cc:1367] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.075s user 0.029s sys 0.000s
I20240503 02:01:00.169342 13169 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.169519 12907 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.169775 16348 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.170156 14866 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.169072 16257 raft_consensus.cc:348] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:00.171053 16257 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.171222 16257 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:00.171629 16219 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.171991 16257 ts_tablet_manager.cc:1398] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:01:00.172385 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-34e88e69beb64036bb676b771dabd2a4 received from {username='slave'} at 127.0.99.133:54909
I20240503 02:01:00.172457 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-34e88e69beb64036bb676b771dabd2a4 on tablet 34e88e69beb64036bb676b771dabd2a4 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.176613 16348 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.179311 13169 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.179716 12907 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.181401 12633 catalog_manager.cc:5484] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.190964 16219 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:00.191689 16219 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.165s user 0.026s sys 0.004s
I20240503 02:01:00.193552 16219 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.193696 16219 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.197510 14934 raft_consensus.cc:1203] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.211865 16219 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.213227 16219 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240503 02:01:00.213891 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-141097686fe24c8f9956dcc1ad1d253c received from {username='slave'} at 127.0.99.135:57047
I20240503 02:01:00.213968 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-141097686fe24c8f9956dcc1ad1d253c on tablet 141097686fe24c8f9956dcc1ad1d253c with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.219357 13169 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.221585 12776 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.222087 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.222347 15883 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.222548 15772 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.240672 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 02:01:00.243057 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04582c3f86d744ccbc96eec50a10994f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.247145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:00.248922 15772 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.249235 12776 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.252452 12633 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.254031 12907 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.258692 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:00.259819 16256 ts_tablet_manager.cc:902] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:00.260252 16256 tablet_copy_client.cc:308] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:00.262274 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 86e314466c01488bb661c596c405db51 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:01:00.262384 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 86e314466c01488bb661c596c405db51 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-86e314466c01488bb661c596c405db51
I20240503 02:01:00.262850 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.263381 13055 tablet_copy_source_session.cc:205] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.266050 16256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.271787 16256 tablet_copy_client.cc:783] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.271840 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.272881 16256 tablet_copy_client.cc:647] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.277271 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:00.278537 16256 tablet_copy_client.cc:523] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.280167 16256 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:00.280618 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.283394 14539 raft_consensus.cc:1203] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.286801 14934 raft_consensus.cc:1203] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.290177 14671 raft_consensus.cc:1203] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.311627 16255 ts_tablet_manager.cc:902] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:00.312110 16255 tablet_copy_client.cc:308] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:00.312551 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 490776bd408a48a9bcbbc7a501baa8ce from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:49001)
I20240503 02:01:00.312656 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 490776bd408a48a9bcbbc7a501baa8ce from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:49001: session id = d23425fa2b5848c8b1a2b737d75e69eb-490776bd408a48a9bcbbc7a501baa8ce
I20240503 02:01:00.313550 12927 tablet_copy_source_session.cc:205] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.314440 16255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.316517 16255 tablet_copy_client.cc:783] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.316674 16255 tablet_copy_client.cc:647] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.329324 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.339000 14671 raft_consensus.cc:1203] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.355437 16255 tablet_copy_client.cc:523] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.355438 14801 raft_consensus.cc:1203] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.355507 16256 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.355789 16256 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:00.356513 16256 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.076s user 0.040s sys 0.006s
I20240503 02:01:00.357079 16255 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:00.357151 16256 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.357314 16256 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.357508 16256 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.357921 16256 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:00.358376 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-86e314466c01488bb661c596c405db51 received from {username='slave'} at 127.0.99.136:51355
I20240503 02:01:00.358462 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-86e314466c01488bb661c596c405db51 on tablet 86e314466c01488bb661c596c405db51 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:00.361279 16219 ts_tablet_manager.cc:902] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:00.361702 16219 tablet_copy_client.cc:308] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:00.373109 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 007feed148fb44e9a6fe9d449b6d8f9d from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40995)
I20240503 02:01:00.373282 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 007feed148fb44e9a6fe9d449b6d8f9d from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40995: session id = 754fc04ea9914073a56c0cf164f63240-007feed148fb44e9a6fe9d449b6d8f9d
I20240503 02:01:00.375524 12796 tablet_copy_source_session.cc:205] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.379285 16256 ts_tablet_manager.cc:902] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:00.379658 16256 tablet_copy_client.cc:308] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:00.380178 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007feed148fb44e9a6fe9d449b6d8f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.380208 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:01:00.380290 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:01:00.381136 13055 tablet_copy_source_session.cc:205] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.382002 16256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.382282 16219 tablet_copy_client.cc:783] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.382416 16219 tablet_copy_client.cc:647] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.384270 16256 tablet_copy_client.cc:783] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.384460 16256 tablet_copy_client.cc:647] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.386188 16219 tablet_copy_client.cc:523] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.387527 16219 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:00.402125 13169 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:00.404531 14801 raft_consensus.cc:1203] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 02:01:00.404588 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04582c3f86d744ccbc96eec50a10994f. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:00.404636 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04582c3f86d744ccbc96eec50a10994f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.404907 12776 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:00.404901 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:00.405426 15772 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:00.405594 16711 ts_tablet_manager.cc:902] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:00.405946 15772 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:00.406225 16711 tablet_copy_client.cc:308] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:00.407224 16256 tablet_copy_client.cc:523] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.409132 16256 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:00.410542 15772 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.414440 12625 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.420048 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 613074d3c4514bd5b64a06c59af09a56 from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:37179)
I20240503 02:01:00.420141 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 613074d3c4514bd5b64a06c59af09a56 from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:37179: session id = d23425fa2b5848c8b1a2b737d75e69eb-613074d3c4514bd5b64a06c59af09a56
I20240503 02:01:00.421023 13055 tablet_copy_source_session.cc:205] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.421952 16711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.422266 12907 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.424023 16711 tablet_copy_client.cc:783] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.424153 16711 tablet_copy_client.cc:647] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.427829 16219 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.428099 16219 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:00.428272 16219 ts_tablet_manager.cc:1367] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.041s user 0.028s sys 0.000s
I20240503 02:01:00.428622 12776 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:00.428791 16219 raft_consensus.cc:348] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.428889 16219 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.429026 16219 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.429280 16219 ts_tablet_manager.cc:1398] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:00.430006 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-007feed148fb44e9a6fe9d449b6d8f9d received from {username='slave'} at 127.0.99.135:40995
I20240503 02:01:00.430095 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-007feed148fb44e9a6fe9d449b6d8f9d on tablet 007feed148fb44e9a6fe9d449b6d8f9d with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.430243 16711 tablet_copy_client.cc:523] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.431490 16711 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:00.436100 12776 consensus_queue.cc:237] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.438577 13169 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:01:00.438796 12710 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ad545d33bbc40458c75ed7c90d92ba7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.438809 16255 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.439056 16255 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:00.439155 16454 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.439218 16255 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.082s user 0.021s sys 0.003s
I20240503 02:01:00.441066 13038 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.442451 14671 raft_consensus.cc:1203] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.443599 15464 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.445259 16257 ts_tablet_manager.cc:902] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:00.445650 16257 tablet_copy_client.cc:308] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:00.445991 16256 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.446309 16256 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:00.446458 16256 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.000s
I20240503 02:01:00.446868 16256 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.446991 16256 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.447113 16256 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.447772 16256 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:01:00.449390 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-ae1f9c8efd2040d6842c6b9f7c6173aa received from {username='slave'} at 127.0.99.136:51355
I20240503 02:01:00.449462 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-ae1f9c8efd2040d6842c6b9f7c6173aa on tablet ae1f9c8efd2040d6842c6b9f7c6173aa with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:00.450443 16734 ts_tablet_manager.cc:902] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:00.450735 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 662782d61c6f472baab4d613b7e6f5cd from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:33089)
I20240503 02:01:00.450856 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 662782d61c6f472baab4d613b7e6f5cd from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:33089: session id = 9e39ab9b64544988b968015fe9fa06e6-662782d61c6f472baab4d613b7e6f5cd
I20240503 02:01:00.451105 16734 tablet_copy_client.cc:308] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:00.452064 12927 tablet_copy_source_session.cc:205] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.452791 15464 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.456390 13169 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.458171 16255 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.458756 16257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 662782d61c6f472baab4d613b7e6f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.458730 12627 catalog_manager.cc:5484] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.460005 13038 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.460687 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 13a95f88ac1c459ba01be0743297f696 from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:44231)
I20240503 02:01:00.460772 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 13a95f88ac1c459ba01be0743297f696 from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:44231: session id = d23425fa2b5848c8b1a2b737d75e69eb-13a95f88ac1c459ba01be0743297f696
I20240503 02:01:00.460893 16257 tablet_copy_client.cc:783] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.461038 16257 tablet_copy_client.cc:647] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.461635 12796 tablet_copy_source_session.cc:205] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.458276 16255 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.463536 16255 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.464730 16255 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20240503 02:01:00.465375 12926 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-490776bd408a48a9bcbbc7a501baa8ce received from {username='slave'} at 127.0.99.134:49001
I20240503 02:01:00.465456 12926 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-490776bd408a48a9bcbbc7a501baa8ce on tablet 490776bd408a48a9bcbbc7a501baa8ce with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:00.466126 16257 tablet_copy_client.cc:523] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.467661 16257 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:00.472116 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.474767 16734 tablet_copy_client.cc:783] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.475032 16734 tablet_copy_client.cc:647] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.478839 16734 tablet_copy_client.cc:523] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.480347 16734 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:00.484898 16711 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.485179 16711 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:00.485342 16711 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.054s user 0.024s sys 0.000s
I20240503 02:01:00.485747 16711 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.485852 16711 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.492039 16711 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.492352 16711 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:01:00.493070 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-613074d3c4514bd5b64a06c59af09a56 received from {username='slave'} at 127.0.99.134:37179
I20240503 02:01:00.493160 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-613074d3c4514bd5b64a06c59af09a56 on tablet 613074d3c4514bd5b64a06c59af09a56 with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:00.526237 16772 ts_tablet_manager.cc:902] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:00.526736 16772 tablet_copy_client.cc:308] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:00.533603 14539 raft_consensus.cc:1203] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.535409 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:00.549916 16734 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.550375 16734 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:00.550634 16734 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.070s user 0.025s sys 0.004s
I20240503 02:01:00.551211 16734 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.551416 16734 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.551666 16734 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.552539 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.552716 16734 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:00.553298 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-13a95f88ac1c459ba01be0743297f696 received from {username='slave'} at 127.0.99.134:44231
I20240503 02:01:00.553375 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-13a95f88ac1c459ba01be0743297f696 on tablet 13a95f88ac1c459ba01be0743297f696 with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:00.553429 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:00.554445 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet f5767bdfcc6249578f7c3ada3c08da6e from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:40631)
I20240503 02:01:00.554548 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet f5767bdfcc6249578f7c3ada3c08da6e from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:40631: session id = 9e39ab9b64544988b968015fe9fa06e6-f5767bdfcc6249578f7c3ada3c08da6e
I20240503 02:01:00.555590 12796 tablet_copy_source_session.cc:205] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.556521 16772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.558815 16772 tablet_copy_client.cc:783] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.559017 16772 tablet_copy_client.cc:647] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.560407 16219 ts_tablet_manager.cc:902] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:00.560848 16219 tablet_copy_client.cc:308] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:00.562201 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 8108933516894d4190ef9a0cf6063be8 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40691)
I20240503 02:01:00.562299 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 8108933516894d4190ef9a0cf6063be8 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40691: session id = 754fc04ea9914073a56c0cf164f63240-8108933516894d4190ef9a0cf6063be8
I20240503 02:01:00.563138 13055 tablet_copy_source_session.cc:205] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.565471 16772 tablet_copy_client.cc:523] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.568081 12776 consensus_queue.cc:237] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.569401 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8108933516894d4190ef9a0cf6063be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.569518 16772 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:00.570256 12907 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.571420 16219 tablet_copy_client.cc:783] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.571385 16451 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.571570 16219 tablet_copy_client.cc:647] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.574231 13169 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.574960 16450 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.577929 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:01:00.578011 12710 consensus_peers.cc:485] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c361f82c69ef477d9c3cfd65e07d7f81. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.584239 16257 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.584498 16257 tablet_bootstrap.cc:492] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:00.584647 16257 ts_tablet_manager.cc:1367] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.117s user 0.023s sys 0.004s
I20240503 02:01:00.585063 16257 raft_consensus.cc:348] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:00.585170 16257 raft_consensus.cc:729] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.585285 16257 consensus_queue.cc:260] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:00.585517 16257 ts_tablet_manager.cc:1398] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:00.586194 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-662782d61c6f472baab4d613b7e6f5cd received from {username='slave'} at 127.0.99.133:33089
I20240503 02:01:00.586277 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-662782d61c6f472baab4d613b7e6f5cd on tablet 662782d61c6f472baab4d613b7e6f5cd with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.595952 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.596330 16219 tablet_copy_client.cc:523] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.597853 16219 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:00.601709 16450 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.604615 12907 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.610080 12630 catalog_manager.cc:5484] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.622745 13169 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.624603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:00.624702 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.630633 16219 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.630914 16219 tablet_bootstrap.cc:492] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:00.631081 16219 ts_tablet_manager.cc:1367] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.000s
I20240503 02:01:00.631492 16219 raft_consensus.cc:348] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.631603 16219 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.631722 16219 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.631999 16219 ts_tablet_manager.cc:1398] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:01:00.632486 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-8108933516894d4190ef9a0cf6063be8 received from {username='slave'} at 127.0.99.135:40691
I20240503 02:01:00.632567 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-8108933516894d4190ef9a0cf6063be8 on tablet 8108933516894d4190ef9a0cf6063be8 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.653143 16772 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.653541 16772 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:00.653777 16772 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.004s
I20240503 02:01:00.654230 16772 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.654422 16772 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.654609 16772 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.658422 16772 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:01:00.659060 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-f5767bdfcc6249578f7c3ada3c08da6e received from {username='slave'} at 127.0.99.133:40631
I20240503 02:01:00.659150 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-f5767bdfcc6249578f7c3ada3c08da6e on tablet f5767bdfcc6249578f7c3ada3c08da6e with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.660730 16219 ts_tablet_manager.cc:902] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:00.661144 16219 tablet_copy_client.cc:308] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:00.661681 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet f5767bdfcc6249578f7c3ada3c08da6e from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40995)
I20240503 02:01:00.661800 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet f5767bdfcc6249578f7c3ada3c08da6e from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40995: session id = 754fc04ea9914073a56c0cf164f63240-f5767bdfcc6249578f7c3ada3c08da6e
I20240503 02:01:00.663091 12796 tablet_copy_source_session.cc:205] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.663992 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.666165 16219 tablet_copy_client.cc:783] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.666749 16219 tablet_copy_client.cc:647] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.669337 16219 tablet_copy_client.cc:523] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.671569 16219 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:00.680199 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:00.702653 16219 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.702940 16219 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:00.703116 16219 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.000s
I20240503 02:01:00.703524 16219 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.703655 16219 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.703801 16219 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.704166 16219 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:01:00.705184 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-f5767bdfcc6249578f7c3ada3c08da6e received from {username='slave'} at 127.0.99.135:40995
I20240503 02:01:00.705276 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-f5767bdfcc6249578f7c3ada3c08da6e on tablet f5767bdfcc6249578f7c3ada3c08da6e with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.711305 12776 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
W20240503 02:01:00.713207 12711 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8292c06377f445849acdc22d08bd612d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.713505 13169 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.714742 13038 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.715411 16466 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.716426 16466 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.732384 16466 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.735147 13038 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.737336 13169 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.739809 12630 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.763494 14671 raft_consensus.cc:1203] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.768527 12776 consensus_queue.cc:237] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:00.770202 12907 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.770313 13038 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.770663 15464 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.771028 15464 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.777244 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:00.777684 14801 raft_consensus.cc:1203] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.787272 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:00.792251 16454 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.794447 13038 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.795258 12630 catalog_manager.cc:5484] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:00.796200 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.796453 12907 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:00.805095 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f3bdd27c750447a87691c0f57a70ee1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.807206 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:00.827425 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.835886 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.836030 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:00.841658 14539 raft_consensus.cc:1203] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.844202 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.849620 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.859335 14934 raft_consensus.cc:1203] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.859594 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:00.863963 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.868345 16219 ts_tablet_manager.cc:902] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:00.871161 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:00.872696 16219 tablet_copy_client.cc:308] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:00.873128 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 6ad545d33bbc40458c75ed7c90d92ba7 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40995)
I20240503 02:01:00.873234 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 6ad545d33bbc40458c75ed7c90d92ba7 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40995: session id = 754fc04ea9914073a56c0cf164f63240-6ad545d33bbc40458c75ed7c90d92ba7
I20240503 02:01:00.874177 12796 tablet_copy_source_session.cc:205] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.875123 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad545d33bbc40458c75ed7c90d92ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.877327 16219 tablet_copy_client.cc:783] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.877475 16219 tablet_copy_client.cc:647] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.892685 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.894145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:00.894704 16219 tablet_copy_client.cc:523] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.896116 16219 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:00.914362 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:00.915073 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:00.927875 16772 ts_tablet_manager.cc:902] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:00.928256 16772 tablet_copy_client.cc:308] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:00.928802 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 04582c3f86d744ccbc96eec50a10994f from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:53947)
I20240503 02:01:00.928905 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 04582c3f86d744ccbc96eec50a10994f from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:53947: session id = 9e39ab9b64544988b968015fe9fa06e6-04582c3f86d744ccbc96eec50a10994f
I20240503 02:01:00.929623 14671 raft_consensus.cc:1203] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.929893 13189 tablet_copy_source_session.cc:205] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:00.931313 16772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:00.933420 16772 tablet_copy_client.cc:783] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:00.933569 16772 tablet_copy_client.cc:647] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:00.933868 14671 raft_consensus.cc:1203] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.947521 12907 consensus_queue.cc:237] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:00.948607 14934 raft_consensus.cc:1203] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:00.949069 13038 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.949548 15196 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:01:00.950029 12842 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da7d3c59eeb949ec84db013f24e656b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.950249 12776 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.950771 15196 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.957785 12774 consensus_queue.cc:237] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.959161 12907 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:01:00.959532 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b829f779c8a459597c47dc3e42a4c46. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:00.959828 16451 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.960368 13037 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:00.960534 14539 raft_consensus.cc:1203] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:00.961642 16450 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:00.962177 16772 tablet_copy_client.cc:523] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:00.963567 16772 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:00.980602 16450 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.981084 13037 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.982594 12907 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.983467 12627 catalog_manager.cc:5484] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index -1 to 4, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:00.989490 16219 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:00.989760 16219 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:00.989919 16219 ts_tablet_manager.cc:1367] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.094s user 0.023s sys 0.000s
I20240503 02:01:00.990486 16219 raft_consensus.cc:348] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.990612 16219 raft_consensus.cc:729] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:00.990705 16982 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.990792 16219 consensus_queue.cc:260] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:00.991658 16219 ts_tablet_manager.cc:1398] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:01:00.992094 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-6ad545d33bbc40458c75ed7c90d92ba7 received from {username='slave'} at 127.0.99.135:40995
I20240503 02:01:00.992161 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-6ad545d33bbc40458c75ed7c90d92ba7 on tablet 6ad545d33bbc40458c75ed7c90d92ba7 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.992992 13038 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.993077 12776 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:00.995666 12630 catalog_manager.cc:5484] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index -1 to 4, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:01.000016 17008 ts_tablet_manager.cc:902] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:01.000698 17008 tablet_copy_client.cc:308] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:01.010707 16772 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.011070 16772 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:01.011361 16772 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.048s user 0.027s sys 0.000s
I20240503 02:01:01.011910 16772 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.012143 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 04582c3f86d744ccbc96eec50a10994f from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:56441)
I20240503 02:01:01.012236 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 04582c3f86d744ccbc96eec50a10994f from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:56441: session id = 558837c679f246dcb3241d66a37d70f1-04582c3f86d744ccbc96eec50a10994f
I20240503 02:01:01.013213 13189 tablet_copy_source_session.cc:205] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.013502 16772 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.014043 17008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.017387 17008 tablet_copy_client.cc:783] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.017539 17008 tablet_copy_client.cc:647] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.018774 17008 tablet_copy_client.cc:523] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.019892 17008 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.023204 16772 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.024299 16772 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:01:01.025846 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-04582c3f86d744ccbc96eec50a10994f received from {username='slave'} at 127.0.99.133:53947
I20240503 02:01:01.025951 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-04582c3f86d744ccbc96eec50a10994f on tablet 04582c3f86d744ccbc96eec50a10994f with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:01.066258 14801 raft_consensus.cc:1203] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:01.067956 14801 raft_consensus.cc:1203] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:01.075214 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.076077 17008 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.076467 17008 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:01.076815 17008 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.057s user 0.028s sys 0.005s
I20240503 02:01:01.077652 17008 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.077804 17008 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.077989 17008 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.078418 17008 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:01:01.079032 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-04582c3f86d744ccbc96eec50a10994f received from {username='slave'} at 127.0.99.136:56441
I20240503 02:01:01.079118 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-04582c3f86d744ccbc96eec50a10994f on tablet 04582c3f86d744ccbc96eec50a10994f with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.080183 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.080394 16219 ts_tablet_manager.cc:902] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.080839 16219 tablet_copy_client.cc:308] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.081743 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet c361f82c69ef477d9c3cfd65e07d7f81 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40995)
I20240503 02:01:01.081857 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet c361f82c69ef477d9c3cfd65e07d7f81 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40995: session id = 754fc04ea9914073a56c0cf164f63240-c361f82c69ef477d9c3cfd65e07d7f81
I20240503 02:01:01.082875 12796 tablet_copy_source_session.cc:205] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.084625 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.085734 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c361f82c69ef477d9c3cfd65e07d7f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.088236 13169 consensus_queue.cc:237] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.089838 13038 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:01:01.090162 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 097947dce87b409ca4a57cb752fdd5e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.090271 16723 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:01.090520 12907 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:01:01.090729 16219 tablet_copy_client.cc:783] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.090889 16219 tablet_copy_client.cc:647] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.091356 15883 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:01:01.095253 16723 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.096484 16219 tablet_copy_client.cc:523] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.098022 16219 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:01.098107 13038 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.098867 12625 catalog_manager.cc:5484] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.101161 12907 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.112527 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.112780 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.114133 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.122206 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.134662 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.143021 12907 consensus_queue.cc:237] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.144760 13038 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.145057 12776 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.145118 16982 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.145371 12842 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da7d3c59eeb949ec84db013f24e656b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.145596 17010 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.146975 16982 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.147382 12776 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.149201 13038 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.149875 12625 catalog_manager.cc:5484] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:01.169242 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da7d3c59eeb949ec84db013f24e656b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.171334 12776 consensus_queue.cc:237] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.172938 13038 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.173323 16451 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.173777 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b829f779c8a459597c47dc3e42a4c46. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:01.173873 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b829f779c8a459597c47dc3e42a4c46. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.175606 12907 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.176054 15464 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.181236 15464 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.181897 12907 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.183501 13038 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.184465 12628 catalog_manager.cc:5484] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.198146 16219 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.198443 16219 tablet_bootstrap.cc:492] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:01.198627 16219 ts_tablet_manager.cc:1367] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.101s user 0.026s sys 0.000s
I20240503 02:01:01.199121 16219 raft_consensus.cc:348] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:01.199261 16219 raft_consensus.cc:729] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.199402 16219 consensus_queue.cc:260] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:01.199723 16219 ts_tablet_manager.cc:1398] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:01.201159 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-c361f82c69ef477d9c3cfd65e07d7f81 received from {username='slave'} at 127.0.99.135:40995
I20240503 02:01:01.201236 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-c361f82c69ef477d9c3cfd65e07d7f81 on tablet c361f82c69ef477d9c3cfd65e07d7f81 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:01.225867 12776 consensus_queue.cc:237] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.228121 13038 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.228940 16451 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.229480 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f3bdd27c750447a87691c0f57a70ee1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.229776 12907 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.230123 16451 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.230401 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f3bdd27c750447a87691c0f57a70ee1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.231848 16451 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.232280 12907 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.234601 13038 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.234691 12629 catalog_manager.cc:5484] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.234855 12776 consensus_queue.cc:237] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
W20240503 02:01:01.237337 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ad545d33bbc40458c75ed7c90d92ba7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.237444 14801 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.237792 13167 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.237982 15464 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.238556 16454 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.240991 16454 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.241585 13038 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.241629 14801 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.241936 16451 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.242847 13167 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.243444 13038 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.244185 12625 catalog_manager.cc:5484] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:01.245555 17173 ts_tablet_manager.cc:902] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.246290 17173 tablet_copy_client.cc:308] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.247161 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 8292c06377f445849acdc22d08bd612d from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:44231)
I20240503 02:01:01.247252 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 8292c06377f445849acdc22d08bd612d from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:44231: session id = d23425fa2b5848c8b1a2b737d75e69eb-8292c06377f445849acdc22d08bd612d
I20240503 02:01:01.248270 12796 tablet_copy_source_session.cc:205] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.249356 17173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.252617 17173 tablet_copy_client.cc:783] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.252846 17173 tablet_copy_client.cc:647] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.254848 17173 tablet_copy_client.cc:523] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.256304 17173 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:01.275652 13167 consensus_queue.cc:237] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
W20240503 02:01:01.277267 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 097947dce87b409ca4a57cb752fdd5e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.277305 13038 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.277731 12907 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.278162 16723 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.278308 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 097947dce87b409ca4a57cb752fdd5e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.278765 16723 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.283134 15883 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:01.285202 13038 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:01.285837 12630 catalog_manager.cc:5484] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:01.288214 12907 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:01.308223 14539 raft_consensus.cc:1203] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:01.325651 17173 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.326124 17173 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:01.327337 17173 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.071s user 0.023s sys 0.003s
I20240503 02:01:01.327934 17173 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:01.328171 17173 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.328647 17173 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:01.340010 17173 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 02:01:01.341313 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-8292c06377f445849acdc22d08bd612d received from {username='slave'} at 127.0.99.134:44231
I20240503 02:01:01.341395 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-8292c06377f445849acdc22d08bd612d on tablet 8292c06377f445849acdc22d08bd612d with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:01.351912 12776 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.354142 14671 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.354521 13167 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.357101 13038 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:01.357764 16466 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.358023 16466 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:01:01.359159 12710 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8292c06377f445849acdc22d08bd612d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:01.359481 15464 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:01.372298 15464 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.374495 13037 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.376736 13167 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.379549 12629 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.391414 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.393652 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.396095 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.404263 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.418426 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.428125 14934 raft_consensus.cc:1203] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:01.431174 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.441857 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.451248 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.452214 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.453023 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.456107 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.456827 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.464324 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.474390 14671 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:01.505138 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.514461 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.515818 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.533385 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.539980 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.547412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.563593 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.574739 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.579833 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.582350 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.588153 17173 ts_tablet_manager.cc:902] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:01.588618 17173 tablet_copy_client.cc:308] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:01.589041 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet da7d3c59eeb949ec84db013f24e656b0 from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:49001)
I20240503 02:01:01.589160 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet da7d3c59eeb949ec84db013f24e656b0 from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:49001: session id = d23425fa2b5848c8b1a2b737d75e69eb-da7d3c59eeb949ec84db013f24e656b0
I20240503 02:01:01.590047 12927 tablet_copy_source_session.cc:205] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.591347 17173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7d3c59eeb949ec84db013f24e656b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.593448 17173 tablet_copy_client.cc:783] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.593595 17173 tablet_copy_client.cc:647] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.595413 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.639742 14802 raft_consensus.cc:1203] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:01:01.640381 17173 tablet_copy_client.cc:523] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.641947 17173 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:01.660173 17415 ts_tablet_manager.cc:902] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.661029 17415 tablet_copy_client.cc:308] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.673645 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 0f3bdd27c750447a87691c0f57a70ee1 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:44493)
I20240503 02:01:01.673746 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 0f3bdd27c750447a87691c0f57a70ee1 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:44493: session id = 558837c679f246dcb3241d66a37d70f1-0f3bdd27c750447a87691c0f57a70ee1
I20240503 02:01:01.674737 12796 tablet_copy_source_session.cc:205] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.676293 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3bdd27c750447a87691c0f57a70ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.681207 17415 tablet_copy_client.cc:783] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.681440 17415 tablet_copy_client.cc:647] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.681478 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.682996 16219 ts_tablet_manager.cc:902] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.683458 16219 tablet_copy_client.cc:308] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.683991 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 3b829f779c8a459597c47dc3e42a4c46 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:40995)
I20240503 02:01:01.684092 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 3b829f779c8a459597c47dc3e42a4c46 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:40995: session id = 754fc04ea9914073a56c0cf164f63240-3b829f779c8a459597c47dc3e42a4c46
I20240503 02:01:01.684927 12796 tablet_copy_source_session.cc:205] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.686158 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b829f779c8a459597c47dc3e42a4c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.691919 16219 tablet_copy_client.cc:783] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.692153 16219 tablet_copy_client.cc:647] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.693696 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.693941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.695039 17415 tablet_copy_client.cc:523] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.697131 17415 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.697966 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.698673 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.703536 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.717060 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.734208 17453 ts_tablet_manager.cc:902] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.734723 17453 tablet_copy_client.cc:308] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.735188 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 3b829f779c8a459597c47dc3e42a4c46 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:44493)
I20240503 02:01:01.735298 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 3b829f779c8a459597c47dc3e42a4c46 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:44493: session id = 558837c679f246dcb3241d66a37d70f1-3b829f779c8a459597c47dc3e42a4c46
I20240503 02:01:01.736299 12796 tablet_copy_source_session.cc:205] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.739068 17453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b829f779c8a459597c47dc3e42a4c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.741568 17453 tablet_copy_client.cc:783] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.741742 17453 tablet_copy_client.cc:647] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.745064 16219 tablet_copy_client.cc:523] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.746632 16219 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:01.752270 17467 ts_tablet_manager.cc:902] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:01.752838 17467 tablet_copy_client.cc:308] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:01.753615 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.776233 16219 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.776526 16219 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:01.776724 16219 ts_tablet_manager.cc:1367] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20240503 02:01:01.777269 16219 raft_consensus.cc:348] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.777402 16219 raft_consensus.cc:729] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.777582 16219 consensus_queue.cc:260] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.777878 16219 ts_tablet_manager.cc:1398] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:01.778445 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-3b829f779c8a459597c47dc3e42a4c46 received from {username='slave'} at 127.0.99.135:40995
I20240503 02:01:01.778524 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-3b829f779c8a459597c47dc3e42a4c46 on tablet 3b829f779c8a459597c47dc3e42a4c46 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:01.785822 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet da7d3c59eeb949ec84db013f24e656b0 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:47631)
I20240503 02:01:01.785959 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet da7d3c59eeb949ec84db013f24e656b0 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:47631: session id = 558837c679f246dcb3241d66a37d70f1-da7d3c59eeb949ec84db013f24e656b0
I20240503 02:01:01.786852 12927 tablet_copy_source_session.cc:205] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.787086 17479 ts_tablet_manager.cc:902] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.787779 17479 tablet_copy_client.cc:308] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.788192 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7d3c59eeb949ec84db013f24e656b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.790037 17467 tablet_copy_client.cc:783] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.790197 17467 tablet_copy_client.cc:647] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.790540 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 0f3bdd27c750447a87691c0f57a70ee1 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:40631)
I20240503 02:01:01.790659 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 0f3bdd27c750447a87691c0f57a70ee1 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:40631: session id = 9e39ab9b64544988b968015fe9fa06e6-0f3bdd27c750447a87691c0f57a70ee1
I20240503 02:01:01.791271 17467 tablet_copy_client.cc:523] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.791321 17448 ts_tablet_manager.cc:902] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.791663 12796 tablet_copy_source_session.cc:205] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.791766 17448 tablet_copy_client.cc:308] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.792593 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3bdd27c750447a87691c0f57a70ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.793480 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 6ad545d33bbc40458c75ed7c90d92ba7 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:40631)
I20240503 02:01:01.793576 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 6ad545d33bbc40458c75ed7c90d92ba7 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:40631: session id = 9e39ab9b64544988b968015fe9fa06e6-6ad545d33bbc40458c75ed7c90d92ba7
I20240503 02:01:01.794116 17467 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.794298 12796 tablet_copy_source_session.cc:205] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.794541 17479 tablet_copy_client.cc:783] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.794673 17479 tablet_copy_client.cc:647] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.796921 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad545d33bbc40458c75ed7c90d92ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.798826 17448 tablet_copy_client.cc:783] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.798969 17448 tablet_copy_client.cc:647] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.800145 17448 tablet_copy_client.cc:523] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.801651 17448 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:01.807828 17503 ts_tablet_manager.cc:902] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:01.808434 17503 tablet_copy_client.cc:308] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:01.808928 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 097947dce87b409ca4a57cb752fdd5e2 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:56441)
I20240503 02:01:01.808961 17453 tablet_copy_client.cc:523] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.809038 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 097947dce87b409ca4a57cb752fdd5e2 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:56441: session id = 558837c679f246dcb3241d66a37d70f1-097947dce87b409ca4a57cb752fdd5e2
I20240503 02:01:01.810055 13189 tablet_copy_source_session.cc:205] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.810225 17453 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.812171 17503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097947dce87b409ca4a57cb752fdd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.812503 17479 tablet_copy_client.cc:523] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.813685 17479 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:01.818380 16219 ts_tablet_manager.cc:902] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:01.818818 16219 tablet_copy_client.cc:308] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:01.819187 17173 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.819267 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 097947dce87b409ca4a57cb752fdd5e2 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:57047)
I20240503 02:01:01.819366 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 097947dce87b409ca4a57cb752fdd5e2 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:57047: session id = 754fc04ea9914073a56c0cf164f63240-097947dce87b409ca4a57cb752fdd5e2
I20240503 02:01:01.819443 17173 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:01.819597 17173 ts_tablet_manager.cc:1367] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.178s user 0.027s sys 0.000s
I20240503 02:01:01.820266 13189 tablet_copy_source_session.cc:205] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.820734 17173 raft_consensus.cc:348] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.820854 17173 raft_consensus.cc:729] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.821771 17173 consensus_queue.cc:260] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.821967 16219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097947dce87b409ca4a57cb752fdd5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.822126 17173 ts_tablet_manager.cc:1398] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:01.822763 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-da7d3c59eeb949ec84db013f24e656b0 received from {username='slave'} at 127.0.99.134:49001
I20240503 02:01:01.822832 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-da7d3c59eeb949ec84db013f24e656b0 on tablet da7d3c59eeb949ec84db013f24e656b0 with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:01.826424 16219 tablet_copy_client.cc:783] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.826561 16219 tablet_copy_client.cc:647] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.837745 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:01.845023 16219 tablet_copy_client.cc:523] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.846510 16219 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:01.850109 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.850345 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:01.866731 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.874346 12756 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:01.883719 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.887343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:01.896896 17479 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.897233 17415 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.897572 17415 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:01.898900 17415 ts_tablet_manager.cc:1367] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.202s user 0.029s sys 0.000s
I20240503 02:01:01.899420 17415 raft_consensus.cc:348] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.899549 17415 raft_consensus.cc:729] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.899744 17415 consensus_queue.cc:260] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.900696 17415 ts_tablet_manager.cc:1398] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:01:01.902043 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-0f3bdd27c750447a87691c0f57a70ee1 received from {username='slave'} at 127.0.99.136:44493
I20240503 02:01:01.902138 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-0f3bdd27c750447a87691c0f57a70ee1 on tablet 0f3bdd27c750447a87691c0f57a70ee1 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.904017 17503 tablet_copy_client.cc:783] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.904246 17503 tablet_copy_client.cc:647] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.907670 17479 tablet_bootstrap.cc:492] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:01.907915 17479 ts_tablet_manager.cc:1367] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.094s user 0.024s sys 0.004s
I20240503 02:01:01.910331 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:01.917193 17479 raft_consensus.cc:348] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.917338 17479 raft_consensus.cc:729] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.917886 17479 consensus_queue.cc:260] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.919584 17479 ts_tablet_manager.cc:1398] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 02:01:01.920408 17448 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.920591 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0f3bdd27c750447a87691c0f57a70ee1 received from {username='slave'} at 127.0.99.133:40631
I20240503 02:01:01.920711 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0f3bdd27c750447a87691c0f57a70ee1 on tablet 0f3bdd27c750447a87691c0f57a70ee1 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:01.922415 17448 tablet_bootstrap.cc:492] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:01.923385 17448 ts_tablet_manager.cc:1367] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.122s user 0.026s sys 0.000s
I20240503 02:01:01.925191 17448 raft_consensus.cc:348] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.925375 17448 raft_consensus.cc:729] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.925547 17448 consensus_queue.cc:260] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:01.925945 17448 ts_tablet_manager.cc:1398] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:01.926817 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-6ad545d33bbc40458c75ed7c90d92ba7 received from {username='slave'} at 127.0.99.133:40631
I20240503 02:01:01.926896 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-6ad545d33bbc40458c75ed7c90d92ba7 on tablet 6ad545d33bbc40458c75ed7c90d92ba7 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:01.910415 17503 tablet_copy_client.cc:523] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.915668 17415 ts_tablet_manager.cc:902] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:01.938134 17415 tablet_copy_client.cc:308] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:01.938633 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet 8292c06377f445849acdc22d08bd612d from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:44493)
I20240503 02:01:01.938741 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet 8292c06377f445849acdc22d08bd612d from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:44493: session id = 558837c679f246dcb3241d66a37d70f1-8292c06377f445849acdc22d08bd612d
I20240503 02:01:01.939812 12796 tablet_copy_source_session.cc:205] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:01.940881 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:01.941843 12776 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60910
I20240503 02:01:01.942095 12776 raft_consensus.cc:595] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:01.943121 17415 tablet_copy_client.cc:783] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:01.943266 17415 tablet_copy_client.cc:647] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:01.944820 17503 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.948457 17415 tablet_copy_client.cc:523] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:01.951050 17415 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:01.947242 17467 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.956614 17467 tablet_bootstrap.cc:492] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:01.956808 17467 ts_tablet_manager.cc:1367] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.163s user 0.027s sys 0.000s
I20240503 02:01:01.930732 17453 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.960182 17453 tablet_bootstrap.cc:492] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:01.960383 17453 ts_tablet_manager.cc:1367] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.150s user 0.027s sys 0.000s
I20240503 02:01:01.960814 17453 raft_consensus.cc:348] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.961005 17453 raft_consensus.cc:729] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.961180 17453 consensus_queue.cc:260] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.961601 17453 ts_tablet_manager.cc:1398] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:01.964298 17467 raft_consensus.cc:348] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.964468 17467 raft_consensus.cc:729] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.964957 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-3b829f779c8a459597c47dc3e42a4c46 received from {username='slave'} at 127.0.99.136:44493
I20240503 02:01:01.965044 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-3b829f779c8a459597c47dc3e42a4c46 on tablet 3b829f779c8a459597c47dc3e42a4c46 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.965520 17467 consensus_queue.cc:260] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.965881 17467 ts_tablet_manager.cc:1398] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 02:01:01.966691 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-da7d3c59eeb949ec84db013f24e656b0 received from {username='slave'} at 127.0.99.136:47631
I20240503 02:01:01.966768 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-da7d3c59eeb949ec84db013f24e656b0 on tablet da7d3c59eeb949ec84db013f24e656b0 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.981969 14671 raft_consensus.cc:1203] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:01.990815 17415 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:01.991132 17415 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:01.991328 17415 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.000s
I20240503 02:01:01.994165 17415 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.994324 17415 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:01.995141 17415 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:01.995486 17415 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:01:01.996031 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-8292c06377f445849acdc22d08bd612d received from {username='slave'} at 127.0.99.136:44493
I20240503 02:01:01.996116 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-8292c06377f445849acdc22d08bd612d on tablet 8292c06377f445849acdc22d08bd612d with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:02.020453 16219 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:02.020805 16219 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:02.021071 16219 ts_tablet_manager.cc:1367] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.175s user 0.031s sys 0.000s
I20240503 02:01:02.021564 16219 raft_consensus.cc:348] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:02.021937 16219 raft_consensus.cc:729] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:02.022092 16219 consensus_queue.cc:260] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:02.022426 16219 ts_tablet_manager.cc:1398] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:02.024178 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-097947dce87b409ca4a57cb752fdd5e2 received from {username='slave'} at 127.0.99.135:57047
I20240503 02:01:02.024268 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-097947dce87b409ca4a57cb752fdd5e2 on tablet 097947dce87b409ca4a57cb752fdd5e2 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:02.027544 16454 raft_consensus.cc:979] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: : Instructing follower 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:02.027642 16454 raft_consensus.cc:1067] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Signalling peer 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:02.028226 13167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
from {username='slave'} at 127.0.99.129:38273
I20240503 02:01:02.028373 13167 raft_consensus.cc:482] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:02.028426 13167 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.029167 13167 raft_consensus.cc:504] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:02.029529 13167 leader_election.cc:288] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:01:02.030128 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
I20240503 02:01:02.030304 14671 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.030306 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:02.030408 12907 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.031316 14671 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:02.031318 12907 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:02.031873 17503 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:02.032256 17503 tablet_bootstrap.cc:492] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:02.031740 13104 leader_election.cc:302] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a, d23425fa2b5848c8b1a2b737d75e69eb; no voters:
I20240503 02:01:02.032423 17092 raft_consensus.cc:2781] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:02.032509 17092 raft_consensus.cc:686] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:02.032506 17503 ts_tablet_manager.cc:1367] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.088s user 0.034s sys 0.000s
I20240503 02:01:02.032656 17092 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:02.032974 17503 raft_consensus.cc:348] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:02.033098 17503 raft_consensus.cc:729] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:02.033249 17503 consensus_queue.cc:260] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:02.033581 17503 ts_tablet_manager.cc:1398] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:01:02.034107 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-097947dce87b409ca4a57cb752fdd5e2 received from {username='slave'} at 127.0.99.136:56441
I20240503 02:01:02.034173 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-097947dce87b409ca4a57cb752fdd5e2 on tablet 097947dce87b409ca4a57cb752fdd5e2 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:02.034981 12625 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 1 to 2, leader changed from bbba999bf95446a68b6586608b33ee16 (127.0.99.129) to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.036175 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:01:02.036304 12776 raft_consensus.cc:3032] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:02.036355 12776 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:01:02.036499 12776 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:02.036679 12776 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.037588 12776 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:02.093942 14933 raft_consensus.cc:1203] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.101840 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.127939 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.135159 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.146910 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.146977 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.147332 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.149438 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.149474 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.157900 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.171909 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.182639 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.186462 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.193354 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.204849 14934 raft_consensus.cc:1203] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.205065 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.206967 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.207876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.211140 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.229447 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.232631 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.232786 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.234582 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.236650 14934 raft_consensus.cc:1203] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.239924 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.247895 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.251570 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.259532 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.274549 14802 raft_consensus.cc:1203] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.283089 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.283430 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.295266 14539 raft_consensus.cc:1203] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.298838 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.302381 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.303413 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.307406 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.314584 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.320667 14802 raft_consensus.cc:1203] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.327502 14934 raft_consensus.cc:1203] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.348870 14539 raft_consensus.cc:1203] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.426164 13037 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:02.427953 14934 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:02.428396 12776 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:02.429267 15274 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.429611 15274 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.432567 15800 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:02.435667 12630 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:02.435593 14934 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:02.444034 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet d86b733f38fd4f08bd40865b99e2e7b2 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:02.444278 12905 consensus_queue.cc:237] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:02.446264 17805 tablet_replica.cc:330] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:02.446420 12774 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:02.446588 14665 raft_consensus.cc:1261] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:02.447052 17010 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.447391 17010 consensus_queue.cc:1035] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.448367 12774 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:02.448482 17805 raft_consensus.cc:2229] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:02.448684 17805 raft_consensus.cc:2258] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:02.449872 17010 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.450134 12776 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.453559 12630 catalog_manager.cc:5484] T b39fa1cb3863414287a47a0832c8b55d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.458420 17805 ts_tablet_manager.cc:1875] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:02.458721 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet b39fa1cb3863414287a47a0832c8b55d with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:02.460979 17805 ts_tablet_manager.cc:1888] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:02.461213 17805 log.cc:1196] T d86b733f38fd4f08bd40865b99e2e7b2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d86b733f38fd4f08bd40865b99e2e7b2
I20240503 02:01:02.462052 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet d86b733f38fd4f08bd40865b99e2e7b2 (table rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]) successfully deleted
I20240503 02:01:02.462849 14665 raft_consensus.cc:2932] T b39fa1cb3863414287a47a0832c8b55d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.468307 17820 tablet_replica.cc:330] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:02.468726 17820 raft_consensus.cc:2229] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:02.468946 17820 raft_consensus.cc:2258] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:02.470319 17820 ts_tablet_manager.cc:1875] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:02.472869 17820 ts_tablet_manager.cc:1888] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:02.472947 17820 log.cc:1196] T b39fa1cb3863414287a47a0832c8b55d P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b39fa1cb3863414287a47a0832c8b55d
I20240503 02:01:02.473526 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet b39fa1cb3863414287a47a0832c8b55d (table rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]) successfully deleted
I20240503 02:01:02.524240 12774 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:02.524922 17092 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.524992 14934 raft_consensus.cc:1203] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:01:02.568399 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.584911 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.587365 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.588331 14665 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:02.588915 17092 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.591039 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.592025 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.602167 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.623829 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.636045 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:02.636123 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.636709 17092 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:02.640187 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.652145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.653039 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.656503 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.667940 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.676184 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.677814 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.686539 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.695353 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.742878 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.745119 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.761528 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.763530 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.778909 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.780103 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.780578 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.782217 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.788625 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.811232 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.817517 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.823151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.833222 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.845294 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.853736 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.906680 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:02.907200 13037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86e314466c01488bb661c596c405db51"
dest_uuid: "22b582a5f4774861a5619c51198d39c5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41438
I20240503 02:01:02.907385 13037 raft_consensus.cc:595] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:02.912895 13016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:02.940275 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:02.942018 14914 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.948843 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.952980 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:02.953747 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:02.960505 15800 raft_consensus.cc:979] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: : Instructing follower 558837c679f246dcb3241d66a37d70f1 to start an election
I20240503 02:01:02.960592 15800 raft_consensus.cc:1067] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Signalling peer 558837c679f246dcb3241d66a37d70f1 to start an election
I20240503 02:01:02.961244 14934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86e314466c01488bb661c596c405db51"
dest_uuid: "558837c679f246dcb3241d66a37d70f1"
from {username='slave'} at 127.0.99.131:45375
I20240503 02:01:02.961444 14934 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:02.961568 14934 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.962647 14934 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:02.965822 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "558837c679f246dcb3241d66a37d70f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:02.965994 13037 raft_consensus.cc:3032] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:02.966053 13037 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:01:02.966238 13037 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:02.966478 13037 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.967262 13169 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:02.967527 13037 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558837c679f246dcb3241d66a37d70f1 in term 2.
I20240503 02:01:02.968616 14665 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:02.968626 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:02.969120 17227 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:02.969317 17092 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:02.971268 17227 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.973278 13169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "558837c679f246dcb3241d66a37d70f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:01:02.973390 13169 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.973389 12906 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.974071 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e314466c01488bb661c596c405db51" candidate_uuid: "558837c679f246dcb3241d66a37d70f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:02.974184 12905 raft_consensus.cc:3037] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:02.974347 13169 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558837c679f246dcb3241d66a37d70f1 in term 2.
I20240503 02:01:02.975020 14869 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5, 558837c679f246dcb3241d66a37d70f1; no voters:
I20240503 02:01:02.975306 16358 raft_consensus.cc:2781] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:02.975378 12905 raft_consensus.cc:2454] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558837c679f246dcb3241d66a37d70f1 in term 2.
I20240503 02:01:02.975414 14934 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [CANDIDATE]: Term 2 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:02.975486 16358 raft_consensus.cc:686] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Becoming Leader. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Running, Role: LEADER
I20240503 02:01:02.975823 16358 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:02.976893 12631 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.978587 14665 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:02.978809 12630 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: term changed from 1 to 2, leader changed from 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) to 558837c679f246dcb3241d66a37d70f1 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:02.984865 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a95f88ac1c459ba01be0743297f696 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:02.988956 18051 tablet_replica.cc:330] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:02.989306 18051 raft_consensus.cc:2229] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:02.989475 18051 raft_consensus.cc:2258] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:02.990988 18051 ts_tablet_manager.cc:1875] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:02.993872 18051 ts_tablet_manager.cc:1888] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:01:02.994033 18051 log.cc:1196] T 13a95f88ac1c459ba01be0743297f696 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/13a95f88ac1c459ba01be0743297f696
I20240503 02:01:02.996651 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 13a95f88ac1c459ba01be0743297f696 (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
I20240503 02:01:03.009373 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.024778 12905 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:03.026963 14665 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.029932 17739 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.030592 13169 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.031070 17010 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.036506 17739 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:03.039196 12630 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:03.039878 13169 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:03.042899 14665 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:03.044385 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:03.044893 18051 tablet_replica.cc:330] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:03.045059 18051 raft_consensus.cc:2229] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.045218 18051 raft_consensus.cc:2258] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.045848 18051 ts_tablet_manager.cc:1875] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.053465 18051 ts_tablet_manager.cc:1888] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.053568 18051 log.cc:1196] T 0613ddd56ed346b3b6dfa9e7c3a4335e P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0613ddd56ed346b3b6dfa9e7c3a4335e
I20240503 02:01:03.054209 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 0613ddd56ed346b3b6dfa9e7c3a4335e (table rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]) successfully deleted
I20240503 02:01:03.059883 13037 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:03.061215 14934 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.061636 12774 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.062304 15800 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.062556 15274 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.067431 15274 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:03.067831 14934 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:03.070950 12631 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:03.072305 12774 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:03.075384 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:03.081406 13167 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:03.082870 14802 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.083529 13037 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.084206 18044 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.084930 17227 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.087576 18139 tablet_replica.cc:330] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:03.087975 18139 raft_consensus.cc:2229] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.088153 18139 raft_consensus.cc:2258] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.089107 18139 ts_tablet_manager.cc:1875] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.090961 18139 ts_tablet_manager.cc:1888] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.091037 18139 log.cc:1196] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:01:03.092263 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet ae1f9c8efd2040d6842c6b9f7c6173aa (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:03.093042 17227 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.093493 13037 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.095856 12625 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.099124 14802 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.104169 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 141097686fe24c8f9956dcc1ad1d253c with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:03.111619 18156 tablet_replica.cc:330] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:03.111868 18156 raft_consensus.cc:2229] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.112123 18156 raft_consensus.cc:2258] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.112776 18156 ts_tablet_manager.cc:1875] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.115337 18156 ts_tablet_manager.cc:1888] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.115479 18156 log.cc:1196] T 141097686fe24c8f9956dcc1ad1d253c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/141097686fe24c8f9956dcc1ad1d253c
I20240503 02:01:03.117345 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 141097686fe24c8f9956dcc1ad1d253c (table rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]) successfully deleted
I20240503 02:01:03.147122 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.149163 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.150677 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.153885 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.155009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.167203 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.185066 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.229441 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.241323 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.242336 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.251063 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.272605 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.275068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.282038 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.298404 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.301162 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.342738 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.353950 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.374456 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.382907 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.392249 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.394695 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.398481 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.406071 13167 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:03.406646 18045 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.411482 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.417397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.419739 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.421154 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.423969 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.431404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.443507 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.443715 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.453194 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.465514 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.467121 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.468470 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.475700 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.483381 13037 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:03.484184 18045 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.487895 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.492439 12905 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:03.492805 18045 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.495261 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.504508 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.530346 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.590757 12774 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:03.592362 14802 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.592768 16454 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.594370 12905 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.594645 16454 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.594947 14802 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.596012 16454 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.598145 12625 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.599362 12905 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:03.615093 13167 consensus_queue.cc:237] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:03.616995 14539 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.617441 13037 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.618131 18044 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.618463 18044 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.618502 12774 consensus_queue.cc:237] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:03.620020 14540 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:03.620380 18390 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:03.621178 14802 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:03.621692 18390 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:03.623318 13038 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:03.625515 16454 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:03.626989 18044 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.629096 13037 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.629788 12634 catalog_manager.cc:5484] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.629925 14539 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.636121 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0633596226954c439d2f671c25eebfe6 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:03.636950 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 007feed148fb44e9a6fe9d449b6d8f9d with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:03.640019 16454 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.642071 14540 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.642061 14802 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.642540 12626 catalog_manager.cc:5484] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.648351 18441 tablet_replica.cc:330] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:03.648681 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad545d33bbc40458c75ed7c90d92ba7 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:03.649026 18441 raft_consensus.cc:2229] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.649370 18441 raft_consensus.cc:2258] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.649981 18441 ts_tablet_manager.cc:1875] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.652503 18440 tablet_replica.cc:330] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:03.652691 18440 raft_consensus.cc:2229] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.652923 18440 raft_consensus.cc:2258] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.653522 18440 ts_tablet_manager.cc:1875] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.653455 13038 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.654414 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.656121 18440 ts_tablet_manager.cc:1888] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.656253 18440 log.cc:1196] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0633596226954c439d2f671c25eebfe6
I20240503 02:01:03.657279 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 0633596226954c439d2f671c25eebfe6 (table rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]) successfully deleted
I20240503 02:01:03.658954 18441 ts_tablet_manager.cc:1888] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.659106 18441 log.cc:1196] T 007feed148fb44e9a6fe9d449b6d8f9d P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/007feed148fb44e9a6fe9d449b6d8f9d
I20240503 02:01:03.659632 18441 tablet_replica.cc:330] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:03.659698 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 007feed148fb44e9a6fe9d449b6d8f9d (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
I20240503 02:01:03.660096 18441 raft_consensus.cc:2229] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.668695 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.671097 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.671825 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.681821 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.683320 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.686751 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.704828 18441 raft_consensus.cc:2258] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.705729 18441 ts_tablet_manager.cc:1875] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.708027 18441 ts_tablet_manager.cc:1888] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:03.713932 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.720660 18441 log.cc:1196] T 6ad545d33bbc40458c75ed7c90d92ba7 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6ad545d33bbc40458c75ed7c90d92ba7
I20240503 02:01:03.721961 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 6ad545d33bbc40458c75ed7c90d92ba7 (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
I20240503 02:01:03.797518 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.798723 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.799423 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.803054 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.811975 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:03.820801 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.831986 13167 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:03.833295 12905 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.833297 14540 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:03.833727 18044 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.834015 18044 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:03.851287 18044 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.854032 14540 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.854382 12630 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.857426 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.860509 12905 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:03.861567 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b734308d3b42958e3a2830a83d44e2 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:03.861793 18440 tablet_replica.cc:330] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:03.861933 18440 raft_consensus.cc:2229] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:03.862088 18440 raft_consensus.cc:2258] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:03.862717 18440 ts_tablet_manager.cc:1875] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:03.864955 18440 ts_tablet_manager.cc:1888] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:03.865025 18440 log.cc:1196] T b4b734308d3b42958e3a2830a83d44e2 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4b734308d3b42958e3a2830a83d44e2
I20240503 02:01:03.870384 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet b4b734308d3b42958e3a2830a83d44e2 (table rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]) successfully deleted
I20240503 02:01:03.893970 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.933329 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.952284 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:03.957679 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.979688 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.982439 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:03.983196 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:03.986464 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:03.998066 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.006359 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.032518 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.035531 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.035656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.036976 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.043124 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.043148 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.043535 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.047120 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.048125 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.069298 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.070286 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.080423 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.084589 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.103708 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.111889 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.152688 12905 consensus_queue.cc:237] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:04.154311 14665 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.154901 14934 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.155869 13038 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.156147 18603 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.156905 18603 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.157099 18677 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.159729 18603 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:04.161875 14665 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:04.162019 14934 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:04.165657 13037 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:04.166102 12625 catalog_manager.cc:5484] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:04.177469 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet da7d3c59eeb949ec84db013f24e656b0 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:04.177840 18440 tablet_replica.cc:330] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:04.177989 18440 raft_consensus.cc:2229] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.178150 18440 raft_consensus.cc:2258] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.178671 18440 ts_tablet_manager.cc:1875] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.181080 18440 ts_tablet_manager.cc:1888] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:04.181144 18440 log.cc:1196] T da7d3c59eeb949ec84db013f24e656b0 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/da7d3c59eeb949ec84db013f24e656b0
I20240503 02:01:04.181655 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet da7d3c59eeb949ec84db013f24e656b0 (table rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]) successfully deleted
I20240503 02:01:04.229981 13167 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:04.230849 13037 consensus_queue.cc:237] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:04.231631 14934 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.232271 12905 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.232370 14802 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.232524 12774 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.232595 18423 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.232797 18433 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.233695 18044 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.234246 18433 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.234983 14540 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.235425 18423 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.237398 18433 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:04.238173 18423 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.238456 14802 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:04.241379 12627 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.241390 12630 catalog_manager.cc:5484] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:04.242043 14934 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.243257 14540 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.246001 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbd697a90534ff3b7a5e17780207d88 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:04.247473 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.248574 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 04582c3f86d744ccbc96eec50a10994f with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:04.248803 18440 tablet_replica.cc:330] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:04.248953 18440 raft_consensus.cc:2229] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.249115 18440 raft_consensus.cc:2258] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.249648 18440 ts_tablet_manager.cc:1875] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.252404 12774 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:04.256708 18440 ts_tablet_manager.cc:1888] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:04.256845 18440 log.cc:1196] T 04582c3f86d744ccbc96eec50a10994f P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/04582c3f86d744ccbc96eec50a10994f
I20240503 02:01:04.257442 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 04582c3f86d744ccbc96eec50a10994f (table rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]) successfully deleted
I20240503 02:01:04.258654 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.260330 18782 tablet_replica.cc:330] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:04.260576 18782 raft_consensus.cc:2229] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.262557 18782 raft_consensus.cc:2258] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.263165 18782 ts_tablet_manager.cc:1875] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.265949 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.274658 18782 ts_tablet_manager.cc:1888] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:04.274924 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.276846 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.279596 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.282482 18782 log.cc:1196] T fdbd697a90534ff3b7a5e17780207d88 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fdbd697a90534ff3b7a5e17780207d88
I20240503 02:01:04.283104 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet fdbd697a90534ff3b7a5e17780207d88 (table rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]) successfully deleted
I20240503 02:01:04.293551 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.307148 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.308591 12905 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.340049 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.345990 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.346184 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.346423 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.350725 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.355937 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.365350 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.382647 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.413357 12905 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:04.414688 14540 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.415127 12774 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.415403 18677 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.415591 18603 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.418480 18083 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.420392 12774 consensus_queue.cc:237] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:04.420688 12776 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.422374 13037 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.422482 14934 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.423380 18390 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.424007 12627 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.426196 16454 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.428010 14539 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:04.428400 18390 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:04.428730 14540 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.429476 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 662782d61c6f472baab4d613b7e6f5cd with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:04.429826 18863 tablet_replica.cc:330] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:04.429960 18863 raft_consensus.cc:2229] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.430109 18863 raft_consensus.cc:2258] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.431506 18863 ts_tablet_manager.cc:1875] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.432865 18390 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.434062 18863 ts_tablet_manager.cc:1888] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:04.435071 14539 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.435217 14934 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.435503 12630 catalog_manager.cc:5484] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.437465 13037 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.441202 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3bdd27c750447a87691c0f57a70ee1 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:04.444475 18863 log.cc:1196] T 662782d61c6f472baab4d613b7e6f5cd P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/662782d61c6f472baab4d613b7e6f5cd
I20240503 02:01:04.446146 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 662782d61c6f472baab4d613b7e6f5cd (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:04.447060 18867 tablet_replica.cc:330] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:04.447286 18867 raft_consensus.cc:2229] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.447517 18867 raft_consensus.cc:2258] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.448102 18867 ts_tablet_manager.cc:1875] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.450349 18867 ts_tablet_manager.cc:1888] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:04.452991 18867 log.cc:1196] T 0f3bdd27c750447a87691c0f57a70ee1 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f3bdd27c750447a87691c0f57a70ee1
I20240503 02:01:04.453636 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 0f3bdd27c750447a87691c0f57a70ee1 (table rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]) successfully deleted
I20240503 02:01:04.482756 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.486557 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.494019 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.500664 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.505163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.508042 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.529517 13037 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:04.531479 12905 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.532009 14539 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:04.532449 18399 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.533211 18399 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:04.537806 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.538159 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.546473 18399 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.548895 12905 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.551128 12628 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.554219 14539 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.555924 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e88e69beb64036bb676b771dabd2a4 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:04.558631 18440 tablet_replica.cc:330] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:04.558774 18440 raft_consensus.cc:2229] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.558923 18440 raft_consensus.cc:2258] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.559455 18440 ts_tablet_manager.cc:1875] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.561661 18440 ts_tablet_manager.cc:1888] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:04.561735 18440 log.cc:1196] T 34e88e69beb64036bb676b771dabd2a4 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/34e88e69beb64036bb676b771dabd2a4
I20240503 02:01:04.562275 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 34e88e69beb64036bb676b771dabd2a4 (table rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]) successfully deleted
I20240503 02:01:04.606324 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.610581 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.612185 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.612399 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.616518 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.635937 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.648283 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.676069 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.690919 12774 consensus_queue.cc:237] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:04.692435 14539 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:04.692823 18866 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:04.694427 18866 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.696122 14539 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.696693 14934 raft_consensus.cc:1261] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:04.698532 18851 consensus_queue.cc:1035] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:04.699911 12626 catalog_manager.cc:5484] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 6 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.702199 14934 raft_consensus.cc:2932] T 0f3bdd27c750447a87691c0f57a70ee1 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:04.711867 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.714316 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3bdd27c750447a87691c0f57a70ee1 with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:04.714601 18863 tablet_replica.cc:330] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:04.714773 18863 raft_consensus.cc:2229] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:04.714988 18863 raft_consensus.cc:2258] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:04.715992 18863 ts_tablet_manager.cc:1875] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:04.718418 18863 ts_tablet_manager.cc:1888] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:01:04.718494 18863 log.cc:1196] T 0f3bdd27c750447a87691c0f57a70ee1 P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0f3bdd27c750447a87691c0f57a70ee1
I20240503 02:01:04.719151 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 0f3bdd27c750447a87691c0f57a70ee1 (table rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]) successfully deleted
I20240503 02:01:04.723099 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.729995 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.739228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.767688 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.776986 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.781332 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.820624 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.856796 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.864828 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.869354 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.878863 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.886726 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.891548 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.892243 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.907577 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.926399 13037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8"
dest_uuid: "22b582a5f4774861a5619c51198d39c5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41694
I20240503 02:01:04.926555 13037 raft_consensus.cc:595] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:04.942380 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.942432 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:04.945039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:04.953678 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:04.960382 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.964449 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:04.984022 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:04.990993 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.043877 12905 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:05.045570 14665 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.046026 18603 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.049461 18603 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.052003 14665 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.054595 12626 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.057912 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:05.059046 12905 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:05.060556 14665 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:05.061118 18603 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:01:05.061992 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0613ddd56ed346b3b6dfa9e7c3a4335e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:05.063205 19153 tablet_replica.cc:330] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:05.063217 18603 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.063560 19153 raft_consensus.cc:2229] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.063789 19153 raft_consensus.cc:2258] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.064465 19153 ts_tablet_manager.cc:1875] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.064975 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:05.065390 14665 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.065806 12629 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.073683 19153 ts_tablet_manager.cc:1888] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:05.076354 19153 log.cc:1196] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0613ddd56ed346b3b6dfa9e7c3a4335e
I20240503 02:01:05.076958 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 0613ddd56ed346b3b6dfa9e7c3a4335e (table rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]) successfully deleted
I20240503 02:01:05.109728 13167 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.110944 14802 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.111390 19122 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.118852 18766 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.121021 14802 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.128628 12627 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.132723 13167 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
W20240503 02:01:05.134378 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 141097686fe24c8f9956dcc1ad1d253c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:05.135659 14802 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:05.136050 18766 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:05.137563 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 141097686fe24c8f9956dcc1ad1d253c with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:05.137756 18863 tablet_replica.cc:330] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:05.137892 18863 raft_consensus.cc:2229] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.138051 18863 raft_consensus.cc:2258] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.138582 18863 ts_tablet_manager.cc:1875] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.140730 18863 ts_tablet_manager.cc:1888] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:05.140782 18863 log.cc:1196] T 141097686fe24c8f9956dcc1ad1d253c P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/141097686fe24c8f9956dcc1ad1d253c
I20240503 02:01:05.141798 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:05.141736 18766 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.143198 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 141097686fe24c8f9956dcc1ad1d253c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:05.144810 14802 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.147485 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 141097686fe24c8f9956dcc1ad1d253c (table rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]) successfully deleted
I20240503 02:01:05.149729 13016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:05.152446 12627 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:05.160037 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.180601 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:05.182951 18399 raft_consensus.cc:979] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: : Instructing follower 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:05.183041 18399 raft_consensus.cc:1067] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Signalling peer 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:05.184016 13169 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
from {username='slave'} at 127.0.99.131:44389
I20240503 02:01:05.184149 13169 raft_consensus.cc:482] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:05.184206 13169 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:05.184692 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.185335 13169 raft_consensus.cc:504] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.185802 13169 leader_election.cc:288] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:01:05.186303 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:05.186420 12905 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:05.186455 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "754fc04ea9914073a56c0cf164f63240"
I20240503 02:01:05.186648 14802 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:05.187384 12905 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:05.187717 14802 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:05.188140 13103 leader_election.cc:302] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 754fc04ea9914073a56c0cf164f63240, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:01:05.188217 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:05.188302 18766 raft_consensus.cc:2781] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:05.188341 13037 raft_consensus.cc:3032] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:05.188362 18766 raft_consensus.cc:686] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:05.188385 13037 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:01:05.188482 18766 consensus_queue.cc:237] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.188536 13037 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.188760 13037 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:05.189798 13037 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:05.192656 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.193413 12627 catalog_manager.cc:5484] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 1 to 2, leader changed from 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.196533 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.205159 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.227223 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.236742 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.237181 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.239782 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.244808 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.246886 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.249612 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.274099 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.363682 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:05.375471 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.378677 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.380012 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.383116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.410109 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.433851 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.434126 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.457604 12905 consensus_queue.cc:237] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.459292 12774 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.459859 18603 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.459919 14802 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.460584 14665 raft_consensus.cc:1261] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.460799 18677 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.461316 18603 consensus_queue.cc:1035] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.466449 18603 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.469779 12632 catalog_manager.cc:5484] T 79c5f35eb8b546de8015c5235913efce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.478308 12774 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.481367 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 79c5f35eb8b546de8015c5235913efce with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:05.481748 18863 tablet_replica.cc:330] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:05.481913 18863 raft_consensus.cc:2229] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.482087 18863 raft_consensus.cc:2258] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.482745 18863 ts_tablet_manager.cc:1875] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.485091 18863 ts_tablet_manager.cc:1888] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:05.485162 18863 log.cc:1196] T 79c5f35eb8b546de8015c5235913efce P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/79c5f35eb8b546de8015c5235913efce
I20240503 02:01:05.485625 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 79c5f35eb8b546de8015c5235913efce (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:05.487169 14665 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.489691 14802 raft_consensus.cc:2932] T 79c5f35eb8b546de8015c5235913efce P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.495493 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.502331 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.513634 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:05.515920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.533262 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.539796 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.547948 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.578102 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.587780 12905 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:05.588248 18766 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.597692 14802 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:05.599138 18766 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.604995 19401 ts_tablet_manager.cc:902] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:05.606298 19401 tablet_copy_client.cc:308] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:05.606712 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 141097686fe24c8f9956dcc1ad1d253c from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:53947)
I20240503 02:01:05.606802 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 141097686fe24c8f9956dcc1ad1d253c from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:53947: session id = 9e39ab9b64544988b968015fe9fa06e6-141097686fe24c8f9956dcc1ad1d253c
I20240503 02:01:05.607729 13189 tablet_copy_source_session.cc:205] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:05.608952 19401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:05.611371 19401 tablet_copy_client.cc:783] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:05.611598 19401 tablet_copy_client.cc:647] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:05.611840 13038 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:01:05.613253 19122 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.625038 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:05.629000 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.630957 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.636521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.638896 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.659160 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.663305 19446 ts_tablet_manager.cc:902] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:05.664047 19446 tablet_copy_client.cc:308] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:05.664729 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:33089)
I20240503 02:01:05.664826 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:33089: session id = 9e39ab9b64544988b968015fe9fa06e6-0613ddd56ed346b3b6dfa9e7c3a4335e
I20240503 02:01:05.665874 12927 tablet_copy_source_session.cc:205] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:05.666664 19401 tablet_copy_client.cc:523] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.667025 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:05.668216 19401 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:05.673756 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.676062 19446 tablet_copy_client.cc:783] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:05.676297 19446 tablet_copy_client.cc:647] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:05.677527 19446 tablet_copy_client.cc:523] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:05.679263 19446 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:05.679361 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.720893 19401 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.721230 19401 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:05.721500 19401 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.053s user 0.024s sys 0.005s
I20240503 02:01:05.722064 19401 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:05.722189 19401 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:05.722321 19401 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:05.723013 19401 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:05.725317 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-141097686fe24c8f9956dcc1ad1d253c received from {username='slave'} at 127.0.99.133:53947
I20240503 02:01:05.725423 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-141097686fe24c8f9956dcc1ad1d253c on tablet 141097686fe24c8f9956dcc1ad1d253c with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:05.740680 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:05.742277 13169 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:05.743203 14933 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:05.743618 19446 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:05.743970 19446 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:05.743937 19271 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.744170 19446 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.065s user 0.021s sys 0.006s
I20240503 02:01:05.744642 19446 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:05.744745 19446 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:05.744885 19446 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:05.745182 19446 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:05.746094 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0613ddd56ed346b3b6dfa9e7c3a4335e received from {username='slave'} at 127.0.99.133:33089
I20240503 02:01:05.746172 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-0613ddd56ed346b3b6dfa9e7c3a4335e on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:05.748175 19271 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.753609 13167 consensus_queue.cc:237] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:05.755484 12905 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.756326 14802 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.756894 14934 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:05.757369 19122 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.758554 18766 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.758956 18766 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:05.759104 19271 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:05.762943 14934 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:05.765120 13169 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:05.767338 12631 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:05.769251 19122 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.771068 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet a105660c12094c71b3a4b80e661d9308 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:05.772660 12905 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.775238 14802 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.778908 12628 catalog_manager.cc:5484] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.781859 14933 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:05.786227 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 097947dce87b409ca4a57cb752fdd5e2 with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:05.787125 18863 tablet_replica.cc:330] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:05.787300 18863 raft_consensus.cc:2229] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.787494 18863 raft_consensus.cc:2258] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.788153 18863 ts_tablet_manager.cc:1875] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.790819 18863 ts_tablet_manager.cc:1888] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:05.790907 18863 log.cc:1196] T 097947dce87b409ca4a57cb752fdd5e2 P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/097947dce87b409ca4a57cb752fdd5e2
I20240503 02:01:05.795578 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 097947dce87b409ca4a57cb752fdd5e2 (table rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]) successfully deleted
I20240503 02:01:05.809700 19491 tablet_replica.cc:330] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:05.809995 19491 raft_consensus.cc:2229] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.810194 19491 raft_consensus.cc:2258] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.810802 19491 ts_tablet_manager.cc:1875] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.813410 19491 ts_tablet_manager.cc:1888] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:05.813575 19491 log.cc:1196] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a105660c12094c71b3a4b80e661d9308
I20240503 02:01:05.818585 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet a105660c12094c71b3a4b80e661d9308 (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:05.854131 13038 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:05.856003 14665 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:05.856369 13169 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:05.857014 19271 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.857337 19271 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:05.877398 19595 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.878926 13169 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.879047 14665 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.883630 12632 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:05.887226 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 613074d3c4514bd5b64a06c59af09a56 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:05.887473 19491 tablet_replica.cc:330] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:05.887634 19491 raft_consensus.cc:2229] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:05.887805 19491 raft_consensus.cc:2258] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:05.888435 19491 ts_tablet_manager.cc:1875] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:05.890594 19491 ts_tablet_manager.cc:1888] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:05.890661 19491 log.cc:1196] T 613074d3c4514bd5b64a06c59af09a56 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/613074d3c4514bd5b64a06c59af09a56
I20240503 02:01:05.891310 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 613074d3c4514bd5b64a06c59af09a56 (table rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]) successfully deleted
I20240503 02:01:05.896428 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:05.904860 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.906625 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.920310 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.927000 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:05.931941 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:05.938324 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.939080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:05.948706 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.950520 14539 raft_consensus.cc:1203] T 141097686fe24c8f9956dcc1ad1d253c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:05.959039 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:05.963742 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:05.991483 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.019296 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.032402 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.037387 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.042544 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.094571 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.094934 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.101826 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.102474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.106145 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.122790 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.123819 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.124120 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.129657 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.135825 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.139218 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.148027 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.185770 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.188021 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.201870 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.222820 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.223799 14539 raft_consensus.cc:1203] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:06.289983 12774 consensus_queue.cc:237] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:06.291360 14934 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.292126 13038 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.292127 18851 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.292644 18851 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.293877 14802 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.294250 19414 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.295935 18851 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.296355 14802 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.298712 14934 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.299337 12628 catalog_manager.cc:5484] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.302412 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b829f779c8a459597c47dc3e42a4c46 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:06.302640 19491 tablet_replica.cc:330] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:06.302765 19491 raft_consensus.cc:2229] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.302913 19491 raft_consensus.cc:2258] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.303422 19491 ts_tablet_manager.cc:1875] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.305413 19491 ts_tablet_manager.cc:1888] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:06.305495 19491 log.cc:1196] T 3b829f779c8a459597c47dc3e42a4c46 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b829f779c8a459597c47dc3e42a4c46
I20240503 02:01:06.305981 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 3b829f779c8a459597c47dc3e42a4c46 (table rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]) successfully deleted
I20240503 02:01:06.310281 13038 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.326200 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.327013 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.342877 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.348453 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.352522 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.359082 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.361129 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.374939 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.379568 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.389860 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.391919 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.392441 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.409878 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.427086 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.438257 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.443737 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.517943 13169 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:06.519656 14802 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.520700 14539 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.521092 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.522099 19431 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.522315 19836 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.522763 19836 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.528016 19847 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:06.532055 14539 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:06.533635 14802 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:06.536465 12629 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:06.538908 13038 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:06.541049 12774 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:06.542922 14934 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.543747 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 063a7f2ed28a455697b000f52a4343a3 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:06.544138 14671 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.544337 18851 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.546780 13169 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:06.547926 19414 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.548301 19414 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:06.556947 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.560086 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.563444 18851 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.566862 14671 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.567381 19935 tablet_replica.cc:330] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:06.567507 19935 raft_consensus.cc:2229] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.568939 12629 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.565817 14934 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.570086 13169 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:06.572409 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8292c06377f445849acdc22d08bd612d with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:06.573468 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.579367 19935 raft_consensus.cc:2258] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.580773 19935 ts_tablet_manager.cc:1875] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.584758 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.584988 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.587273 19935 ts_tablet_manager.cc:1888] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:06.591744 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.593845 19935 log.cc:1196] T 063a7f2ed28a455697b000f52a4343a3 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/063a7f2ed28a455697b000f52a4343a3
I20240503 02:01:06.599231 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.600023 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 063a7f2ed28a455697b000f52a4343a3 (table rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]) successfully deleted
I20240503 02:01:06.613874 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.626713 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.638331 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.643358 19970 tablet_replica.cc:330] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:06.643816 19970 raft_consensus.cc:2229] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.644024 19970 raft_consensus.cc:2258] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.644613 19970 ts_tablet_manager.cc:1875] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.652010 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.653229 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.666896 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.669965 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.678355 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.678356 19970 ts_tablet_manager.cc:1888] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:06.678432 19970 log.cc:1196] T 8292c06377f445849acdc22d08bd612d P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8292c06377f445849acdc22d08bd612d
I20240503 02:01:06.681563 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.685672 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.690989 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.703817 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 8292c06377f445849acdc22d08bd612d (table rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]) successfully deleted
I20240503 02:01:06.715996 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.721729 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.722412 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.732015 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.732061 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.754463 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.876403 13169 consensus_queue.cc:237] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:06.878396 14665 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:06.878839 19431 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:06.879338 12907 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:06.883129 19431 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:06.895124 19431 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:06.897473 12907 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:06.900425 12625 catalog_manager.cc:5484] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:06.903342 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet e6cc67c73560449e81303a948df8f9df with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:06.903728 19970 tablet_replica.cc:330] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:06.903880 19970 raft_consensus.cc:2229] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:06.904033 19970 raft_consensus.cc:2258] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:06.904655 19970 ts_tablet_manager.cc:1875] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:06.906916 19970 ts_tablet_manager.cc:1888] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:06.906986 19970 log.cc:1196] T e6cc67c73560449e81303a948df8f9df P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6cc67c73560449e81303a948df8f9df
I20240503 02:01:06.907780 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet e6cc67c73560449e81303a948df8f9df (table rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]) successfully deleted
I20240503 02:01:06.918066 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.919629 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.924126 14665 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:06.933357 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:06.934029 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.947695 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.959270 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.960959 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:06.971907 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:06.975317 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:06.977357 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.984033 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.994935 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:06.997582 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.000511 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.003942 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.010675 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.037412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.040903 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.078660 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.082993 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.089116 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.095257 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.096683 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.099567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.104408 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.138880 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.143596 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.169312 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.171662 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.180070 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.182430 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.194665 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.228420 14934 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:07.229926 13167 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:07.231184 20259 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:07.231467 13038 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:07.233058 18329 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:07.235837 20259 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.236069 13167 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.238835 12629 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.245214 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e314466c01488bb661c596c405db51 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:07.245544 13038 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.269981 20287 tablet_replica.cc:330] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:07.270466 20287 raft_consensus.cc:2229] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:07.272938 20287 raft_consensus.cc:2258] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:07.273451 20287 ts_tablet_manager.cc:1875] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:07.280150 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.282817 20287 ts_tablet_manager.cc:1888] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:01:07.282896 20287 log.cc:1196] T 86e314466c01488bb661c596c405db51 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/86e314466c01488bb661c596c405db51
I20240503 02:01:07.283425 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 86e314466c01488bb661c596c405db51 (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:07.291401 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.299191 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.303233 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.309422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.313383 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.323122 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.324748 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.345428 13038 consensus_queue.cc:237] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:07.346913 13167 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.346961 14934 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.347375 19975 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.347676 19975 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.349216 19975 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.351119 13167 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.353227 12634 catalog_manager.cc:5484] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.357224 14934 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:07.357883 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c464b6e30bb414bbe70ed030c5f481c with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:07.358108 20287 tablet_replica.cc:330] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:07.358243 20287 raft_consensus.cc:2229] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:07.358382 20287 raft_consensus.cc:2258] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:07.359011 20287 ts_tablet_manager.cc:1875] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:07.361339 20287 ts_tablet_manager.cc:1888] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:07.361389 20287 log.cc:1196] T 1c464b6e30bb414bbe70ed030c5f481c P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c464b6e30bb414bbe70ed030c5f481c
I20240503 02:01:07.362514 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 1c464b6e30bb414bbe70ed030c5f481c (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:07.409284 12774 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:07.410992 14539 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:07.411128 14802 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:07.411530 19966 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:07.411696 19794 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:07.411789 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:07.412178 19794 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:07.414351 19794 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.414758 13038 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.418571 12629 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.421573 14802 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.422847 14539 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.424346 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet f5767bdfcc6249578f7c3ada3c08da6e with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:07.426256 20427 tablet_replica.cc:330] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:07.427137 20427 raft_consensus.cc:2229] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:07.427953 20427 raft_consensus.cc:2258] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:07.428499 20427 ts_tablet_manager.cc:1875] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:07.430855 20427 ts_tablet_manager.cc:1888] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:07.431514 20427 log.cc:1196] T f5767bdfcc6249578f7c3ada3c08da6e P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f5767bdfcc6249578f7c3ada3c08da6e
I20240503 02:01:07.432147 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet f5767bdfcc6249578f7c3ada3c08da6e (table rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]) successfully deleted
I20240503 02:01:07.466023 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.471240 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.473438 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.478721 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.486466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.512569 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.516374 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.524484 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.537478 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.541900 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.546924 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.560791 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.570092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.591462 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.595163 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.600540 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.656459 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.656538 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.656550 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.672109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.675294 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.677037 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.693598 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.696230 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.707005 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.726204 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.726776 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.736323 14519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.740576 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.743237 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.769078 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.770119 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.793107 12774 consensus_queue.cc:237] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:07.793668 13167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54598
I20240503 02:01:07.793828 13167 raft_consensus.cc:595] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:07.794741 12907 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.795385 14802 raft_consensus.cc:1261] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.796020 20412 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.797329 20412 consensus_queue.cc:1035] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.802896 20412 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.803470 12907 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.806934 12634 catalog_manager.cc:5484] T c361f82c69ef477d9c3cfd65e07d7f81 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.811287 14802 raft_consensus.cc:2932] T c361f82c69ef477d9c3cfd65e07d7f81 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.812279 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet c361f82c69ef477d9c3cfd65e07d7f81 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:07.812510 20427 tablet_replica.cc:330] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:07.812650 20427 raft_consensus.cc:2229] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:07.812789 20427 raft_consensus.cc:2258] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:07.813457 20427 ts_tablet_manager.cc:1875] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:07.815748 20427 ts_tablet_manager.cc:1888] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:07.817636 20427 log.cc:1196] T c361f82c69ef477d9c3cfd65e07d7f81 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c361f82c69ef477d9c3cfd65e07d7f81
I20240503 02:01:07.818897 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet c361f82c69ef477d9c3cfd65e07d7f81 (table rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]) successfully deleted
I20240503 02:01:07.844237 12774 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42082
I20240503 02:01:07.844378 12774 raft_consensus.cc:595] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:07.859455 12907 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:07.860775 13167 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.861508 14665 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:01:07.861877 20005 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.863745 20412 raft_consensus.cc:979] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: : Instructing follower 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:07.863869 19794 raft_consensus.cc:1067] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Signalling peer 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:07.864331 14539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e"
dest_uuid: "9e39ab9b64544988b968015fe9fa06e6"
from {username='slave'} at 127.0.99.129:52133
I20240503 02:01:07.864522 14539 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.864640 14539 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.864679 20005 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:01:07.865607 14539 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:07.866851 14539 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:01:07.867116 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:07.867233 13038 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.868278 13038 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:07.868523 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:01:07.868664 12774 raft_consensus.cc:3032] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.868705 12774 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:01:07.868870 12774 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:07.869100 12774 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.870004 12774 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:07.870990 14474 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 9e39ab9b64544988b968015fe9fa06e6, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:01:07.871204 20428 raft_consensus.cc:2781] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.871366 20428 raft_consensus.cc:686] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Becoming Leader. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Running, Role: LEADER
I20240503 02:01:07.871693 20428 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:07.874035 12627 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: term changed from 1 to 2, leader changed from bbba999bf95446a68b6586608b33ee16 (127.0.99.129) to 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:07.877561 13149 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:01:07.880368 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:07.881561 14782 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.882715 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.888072 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.891981 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:07.900600 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:07.910405 19920 raft_consensus.cc:979] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: : Instructing follower 754fc04ea9914073a56c0cf164f63240 to start an election
I20240503 02:01:07.910508 19920 raft_consensus.cc:1067] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Signalling peer 754fc04ea9914073a56c0cf164f63240 to start an election
I20240503 02:01:07.911029 14802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3"
dest_uuid: "754fc04ea9914073a56c0cf164f63240"
from {username='slave'} at 127.0.99.132:35815
I20240503 02:01:07.911216 14802 raft_consensus.cc:482] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:07.911288 14802 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.912381 14802 raft_consensus.cc:504] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:07.914374 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:07.914506 13038 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.915428 13038 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:07.915956 14802 leader_election.cc:288] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 2 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:01:07.917577 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:01:07.917706 13167 raft_consensus.cc:3032] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:07.917749 13167 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:07.917909 13167 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:07.918112 13167 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.919035 13167 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:07.919361 14737 leader_election.cc:302] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 22b582a5f4774861a5619c51198d39c5, 754fc04ea9914073a56c0cf164f63240; no voters:
I20240503 02:01:07.919605 20426 raft_consensus.cc:2781] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:07.919821 20426 raft_consensus.cc:686] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Becoming Leader. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Running, Role: LEADER
I20240503 02:01:07.920147 20426 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:07.921732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:07.925084 12627 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: term changed from 1 to 2, leader changed from 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) to 754fc04ea9914073a56c0cf164f63240 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:07.934100 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "754fc04ea9914073a56c0cf164f63240"
I20240503 02:01:07.934216 14802 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.935163 14802 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:07.937251 20005 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:07.943619 12627 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:07.944730 13167 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:07.959815 14665 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:07.964591 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063a7f2ed28a455697b000f52a4343a3" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "9e39ab9b64544988b968015fe9fa06e6"
I20240503 02:01:07.964784 14539 raft_consensus.cc:3037] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:07.965951 14539 raft_consensus.cc:2454] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:07.966018 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 490776bd408a48a9bcbbc7a501baa8ce with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:07.975625 20718 tablet_replica.cc:330] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:07.975826 20718 raft_consensus.cc:2229] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:07.975991 20718 raft_consensus.cc:2258] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:07.976908 20718 ts_tablet_manager.cc:1875] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:07.979321 20718 ts_tablet_manager.cc:1888] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:01:07.979393 20718 log.cc:1196] T 490776bd408a48a9bcbbc7a501baa8ce P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/490776bd408a48a9bcbbc7a501baa8ce
I20240503 02:01:07.980638 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 490776bd408a48a9bcbbc7a501baa8ce (table rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]) successfully deleted
I20240503 02:01:07.997308 13038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa"
dest_uuid: "22b582a5f4774861a5619c51198d39c5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46262
I20240503 02:01:07.997470 13038 raft_consensus.cc:595] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:08.035902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.036634 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.037391 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.038318 13016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.042737 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.048218 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.053556 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.062319 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.064227 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.067879 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.070310 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.083921 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.104218 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.110935 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.118063 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.136894 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.176633 20355 raft_consensus.cc:979] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: : Instructing follower bbba999bf95446a68b6586608b33ee16 to start an election
I20240503 02:01:08.176746 20355 raft_consensus.cc:1067] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Signalling peer bbba999bf95446a68b6586608b33ee16 to start an election
I20240503 02:01:08.177218 12774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
from {username='slave'} at 127.0.99.131:50139
I20240503 02:01:08.177378 12774 raft_consensus.cc:482] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:08.177456 12774 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.178550 12774 raft_consensus.cc:504] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:08.178965 12774 leader_election.cc:288] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 2 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:01:08.179570 14934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "558837c679f246dcb3241d66a37d70f1"
I20240503 02:01:08.179774 14934 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.180794 14934 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 2.
I20240503 02:01:08.181514 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "bbba999bf95446a68b6586608b33ee16" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:08.181646 13038 raft_consensus.cc:3032] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:08.181687 13038 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:01:08.181828 13038 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:08.181964 13038 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.182958 13038 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbba999bf95446a68b6586608b33ee16 in term 2.
I20240503 02:01:08.183446 12710 leader_election.cc:302] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 558837c679f246dcb3241d66a37d70f1, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:01:08.184108 19794 raft_consensus.cc:2781] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:08.184213 19794 raft_consensus.cc:686] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Becoming Leader. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:01:08.184362 19794 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:08.187572 12630 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: term changed from 1 to 2, leader changed from 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) to bbba999bf95446a68b6586608b33ee16 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:08.208969 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.217948 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.236861 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.241268 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.244100 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.250478 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.253875 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.270359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.299963 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.309381 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.310988 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.335790 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.340135 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.350203 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.360177 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.360416 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.366663 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.377830 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.379191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.387871 12776 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.388473 20668 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.399322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.399621 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.399747 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.400980 13167 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.401717 20689 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.407452 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.434276 14802 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.434715 20668 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.439204 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.439615 20689 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.454510 14539 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.459411 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.460367 20956 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.460834 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.463840 20689 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.472134 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.476801 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.477778 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.485391 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.490150 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.500365 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.500926 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.515095 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.534235 12776 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42170
I20240503 02:01:08.534399 12776 raft_consensus.cc:595] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:08.565012 14934 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:08.566442 13038 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:08.567693 18329 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:08.569783 13167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54692
I20240503 02:01:08.569921 13167 raft_consensus.cc:595] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:08.570034 18329 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:08.570310 13038 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:08.572834 12626 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:08.576519 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e314466c01488bb661c596c405db51 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:08.578871 21035 tablet_replica.cc:330] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:08.579130 21035 raft_consensus.cc:2229] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:08.579361 21035 raft_consensus.cc:2258] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:08.580739 21035 ts_tablet_manager.cc:1875] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:08.582953 21035 ts_tablet_manager.cc:1888] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:01:08.583025 21035 log.cc:1196] T 86e314466c01488bb661c596c405db51 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/86e314466c01488bb661c596c405db51
I20240503 02:01:08.583606 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 86e314466c01488bb661c596c405db51 (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:08.590082 14934 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:08.592157 13038 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:08.592571 18329 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:08.594971 18329 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:08.595690 13038 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:08.596603 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 86e314466c01488bb661c596c405db51 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 02:01:08.600102 14868 consensus_peers.cc:485] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86e314466c01488bb661c596c405db51. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:08.600261 12626 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 7 to 8, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:08.676298 14934 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:08.676731 20412 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:08.679733 13038 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:08.681181 19794 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:08.692503 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.696494 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.697079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.711514 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.713040 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.723116 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.726320 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.736939 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.759495 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.769999 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.773643 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.784055 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.784552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.784969 14782 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.787264 12756 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.812026 19794 raft_consensus.cc:979] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: : Instructing follower 754fc04ea9914073a56c0cf164f63240 to start an election
I20240503 02:01:08.812140 19794 raft_consensus.cc:1067] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Signalling peer 754fc04ea9914073a56c0cf164f63240 to start an election
I20240503 02:01:08.812551 14802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d"
dest_uuid: "754fc04ea9914073a56c0cf164f63240"
from {username='slave'} at 127.0.99.129:36657
I20240503 02:01:08.812698 14802 raft_consensus.cc:482] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:08.812757 14802 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.813825 14802 raft_consensus.cc:504] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:08.831507 14802 leader_election.cc:288] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 2 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:08.832013 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:01:08.832159 12776 raft_consensus.cc:3032] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:08.832134 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007feed148fb44e9a6fe9d449b6d8f9d" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:08.832203 12776 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:01:08.832257 12905 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.832350 12776 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:08.832547 12776 raft_consensus.cc:3037] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:08.833341 12905 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:08.833523 12776 raft_consensus.cc:2454] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:08.833731 14737 leader_election.cc:302] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:01:08.833931 20948 raft_consensus.cc:2781] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:08.834040 20948 raft_consensus.cc:686] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Becoming Leader. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Running, Role: LEADER
I20240503 02:01:08.834218 20948 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:08.836254 12626 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 reported cstate change: term changed from 1 to 2, leader changed from bbba999bf95446a68b6586608b33ee16 (127.0.99.129) to 754fc04ea9914073a56c0cf164f63240 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:08.844415 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.860397 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:08.869937 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.875560 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:08.879554 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.890367 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.892839 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.893641 14651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:01:08.911903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.946265 19920 raft_consensus.cc:979] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: : Instructing follower d23425fa2b5848c8b1a2b737d75e69eb to start an election
I20240503 02:01:08.946372 19920 raft_consensus.cc:1067] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Signalling peer d23425fa2b5848c8b1a2b737d75e69eb to start an election
I20240503 02:01:08.946923 14665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696"
dest_uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
from {username='slave'} at 127.0.99.132:58521
I20240503 02:01:08.947114 14665 raft_consensus.cc:482] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:08.947178 14665 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.948105 14665 raft_consensus.cc:504] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:08.951239 14665 leader_election.cc:288] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [CANDIDATE]: Term 3 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:08.951689 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:01:08.951853 13167 raft_consensus.cc:3032] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:08.951906 13167 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:08.952061 13167 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:08.952255 13167 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.952689 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a95f88ac1c459ba01be0743297f696" candidate_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:08.952824 12905 raft_consensus.cc:3037] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:08.955240 12905 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23425fa2b5848c8b1a2b737d75e69eb in term 3.
I20240503 02:01:08.955297 13167 raft_consensus.cc:2454] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d23425fa2b5848c8b1a2b737d75e69eb in term 3.
I20240503 02:01:08.955842 14606 leader_election.cc:302] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, d23425fa2b5848c8b1a2b737d75e69eb; no voters:
I20240503 02:01:08.958956 21274 raft_consensus.cc:2781] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:08.959103 21274 raft_consensus.cc:686] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 LEADER]: Becoming Leader. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Running, Role: LEADER
I20240503 02:01:08.959390 21274 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:08.964097 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:08.965709 12634 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: term changed from 2 to 3, leader changed from 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) to d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134). New cstate: current_term: 3 leader_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:08.978037 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:08.978325 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.985818 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:08.991925 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:08.996090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.004379 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.022094 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.050997 21311 ts_tablet_manager.cc:902] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:01:09.051646 21311 tablet_copy_client.cc:308] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41447
I20240503 02:01:09.062263 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.063545 14954 tablet_copy_service.cc:140] P 558837c679f246dcb3241d66a37d70f1: Received BeginTabletCopySession request for tablet 86e314466c01488bb661c596c405db51 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:35607)
I20240503 02:01:09.063685 14954 tablet_copy_service.cc:161] P 558837c679f246dcb3241d66a37d70f1: Beginning new tablet copy session on tablet 86e314466c01488bb661c596c405db51 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:35607: session id = 754fc04ea9914073a56c0cf164f63240-86e314466c01488bb661c596c405db51
I20240503 02:01:09.065106 14954 tablet_copy_source_session.cc:205] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:09.065423 13038 consensus_queue.cc:237] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:09.066916 14934 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:09.067600 20355 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.069726 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:09.076668 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.080998 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.083958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.088476 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.092311 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.098826 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.110369 21311 tablet_copy_client.cc:783] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:09.110564 21311 tablet_copy_client.cc:647] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:09.119616 21311 tablet_copy_client.cc:523] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.120186 20355 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.123337 12627 catalog_manager.cc:5484] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.123492 14934 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.128966 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c464b6e30bb414bbe70ed030c5f481c with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:09.129582 13038 consensus_queue.cc:237] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.130228 21311 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:09.131551 14934 raft_consensus.cc:1261] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:09.131997 19975 consensus_queue.cc:1035] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:01:09.133203 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c464b6e30bb414bbe70ed030c5f481c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:09.133236 21362 tablet_replica.cc:330] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:09.133450 21362 raft_consensus.cc:2229] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:09.133687 21362 raft_consensus.cc:2258] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:09.134287 21362 ts_tablet_manager.cc:1875] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:09.134362 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.135159 19975 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.137943 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1c464b6e30bb414bbe70ed030c5f481c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:09.138322 14934 raft_consensus.cc:2932] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.141705 12630 catalog_manager.cc:5484] T 1c464b6e30bb414bbe70ed030c5f481c P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.146212 21362 ts_tablet_manager.cc:1888] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:09.146361 21362 log.cc:1196] T 1c464b6e30bb414bbe70ed030c5f481c P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c464b6e30bb414bbe70ed030c5f481c
I20240503 02:01:09.146991 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 1c464b6e30bb414bbe70ed030c5f481c (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:09.211222 21311 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.211565 21311 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:09.211738 21311 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.082s user 0.028s sys 0.000s
I20240503 02:01:09.212213 21311 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:09.212329 21311 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:09.215368 21311 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:09.215808 21311 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:01:09.216449 14954 tablet_copy_service.cc:342] P 558837c679f246dcb3241d66a37d70f1: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-86e314466c01488bb661c596c405db51 received from {username='slave'} at 127.0.99.135:35607
I20240503 02:01:09.216545 14954 tablet_copy_service.cc:433] P 558837c679f246dcb3241d66a37d70f1: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-86e314466c01488bb661c596c405db51 on tablet 86e314466c01488bb661c596c405db51 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:09.227593 13167 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.230986 14539 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:09.233597 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.234050 21261 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.237205 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.247500 21261 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.248061 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.249255 14802 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.250497 12629 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.250990 14540 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:09.251022 14539 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.251521 20948 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:09.253254 20948 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.255724 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.256314 12634 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.257194 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b734308d3b42958e3a2830a83d44e2 with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:09.261482 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:09.262030 14540 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.266041 20960 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:09.266826 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:09.266916 13167 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:09.268112 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.268208 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 063a7f2ed28a455697b000f52a4343a3 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:09.268404 21362 tablet_replica.cc:330] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:09.268543 21362 raft_consensus.cc:2229] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:09.268666 21362 raft_consensus.cc:2258] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:09.268942 20960 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.269544 21362 ts_tablet_manager.cc:1875] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:09.269944 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:09.272938 13103 consensus_peers.cc:485] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4b734308d3b42958e3a2830a83d44e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:09.273028 14540 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:09.273141 21362 ts_tablet_manager.cc:1888] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:01:09.273200 21362 log.cc:1196] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/063a7f2ed28a455697b000f52a4343a3
I20240503 02:01:09.273659 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 063a7f2ed28a455697b000f52a4343a3 (table rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]) successfully deleted
I20240503 02:01:09.274062 21262 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:09.273999 13038 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.277448 21262 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.280400 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b4b734308d3b42958e3a2830a83d44e2 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:09.280997 14540 raft_consensus.cc:2932] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.281157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.281565 12627 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:09.286234 12776 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:09.287046 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.287828 20960 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.289121 21462 tablet_replica.cc:330] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:09.290661 21462 raft_consensus.cc:2229] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:09.290910 21462 raft_consensus.cc:2258] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:09.291532 21462 ts_tablet_manager.cc:1875] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:09.300441 21462 ts_tablet_manager.cc:1888] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:09.301384 21462 log.cc:1196] T b4b734308d3b42958e3a2830a83d44e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b4b734308d3b42958e3a2830a83d44e2
I20240503 02:01:09.301965 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet b4b734308d3b42958e3a2830a83d44e2 (table rebalance_test_table_0 [id=8a53b815c07146bf988000c73f7e70d4]) successfully deleted
I20240503 02:01:09.352205 12905 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd"
dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55502
I20240503 02:01:09.352401 12905 raft_consensus.cc:595] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:09.373418 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.373792 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.377146 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.392686 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.396773 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.414216 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.422782 14519 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.429163 20694 raft_consensus.cc:979] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: : Instructing follower 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:09.429244 20694 raft_consensus.cc:1067] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Signalling peer 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:09.430007 14540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd"
dest_uuid: "9e39ab9b64544988b968015fe9fa06e6"
from {username='slave'} at 127.0.99.130:53243
I20240503 02:01:09.430150 14540 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:09.430207 14540 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.431182 14540 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.431596 14540 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:09.432291 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:01:09.432428 12776 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.432500 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "662782d61c6f472baab4d613b7e6f5cd" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:09.432608 12905 raft_consensus.cc:3032] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:09.432658 12905 raft_consensus.cc:729] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:01:09.432808 12905 consensus_queue.cc:260] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.432960 12905 raft_consensus.cc:3037] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:09.433497 12776 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:09.433964 12905 raft_consensus.cc:2454] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:09.433980 14474 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e39ab9b64544988b968015fe9fa06e6, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:01:09.434140 20958 raft_consensus.cc:2781] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:09.434273 20958 raft_consensus.cc:686] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Becoming Leader. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Running, Role: LEADER
I20240503 02:01:09.434389 20958 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.436108 12627 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: term changed from 1 to 2, leader changed from 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) to 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:09.440932 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.467406 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:01:09.468196 21611 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:09.480484 13167 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:01:09.481779 21611 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:09.524534 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.531663 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.533627 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.539963 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.541404 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:09.548130 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.551534 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.552887 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.582772 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.592578 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.593585 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.602025 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:09.603356 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.607184 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.613750 13038 consensus_queue.cc:237] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:09.614851 14802 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:09.615258 20355 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.633788 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.634856 14801 raft_consensus.cc:1203] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:01:09.644131 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.650725 21352 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:09.653077 14802 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:09.654011 12628 catalog_manager.cc:5484] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 6, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:09.655380 21721 ts_tablet_manager.cc:902] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:09.655889 21721 tablet_copy_client.cc:308] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:09.656327 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet b4b734308d3b42958e3a2830a83d44e2 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:56441)
I20240503 02:01:09.656424 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet b4b734308d3b42958e3a2830a83d44e2 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:56441: session id = 558837c679f246dcb3241d66a37d70f1-b4b734308d3b42958e3a2830a83d44e2
I20240503 02:01:09.657307 13189 tablet_copy_source_session.cc:205] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:09.661691 21721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b734308d3b42958e3a2830a83d44e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:09.661749 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbd697a90534ff3b7a5e17780207d88 with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:09.663691 13038 consensus_queue.cc:237] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:09.665167 14802 raft_consensus.cc:1261] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:01:09.665755 12973 consensus_peers.cc:485] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdbd697a90534ff3b7a5e17780207d88. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:09.665891 21352 consensus_queue.cc:1035] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:09.667394 21352 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:09.668215 14802 raft_consensus.cc:2932] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:09.669780 21721 tablet_copy_client.cc:783] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:09.669929 21721 tablet_copy_client.cc:647] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:09.671041 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fdbd697a90534ff3b7a5e17780207d88 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:09.672051 12632 catalog_manager.cc:5484] T fdbd697a90534ff3b7a5e17780207d88 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 6 to 7, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:09.682826 21732 tablet_replica.cc:330] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:09.683019 21732 raft_consensus.cc:2229] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:09.683218 21732 raft_consensus.cc:2258] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:09.687811 21732 ts_tablet_manager.cc:1875] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:09.689253 21721 tablet_copy_client.cc:523] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.690660 21732 ts_tablet_manager.cc:1888] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:09.690702 21721 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:09.690740 21732 log.cc:1196] T fdbd697a90534ff3b7a5e17780207d88 P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdbd697a90534ff3b7a5e17780207d88
I20240503 02:01:09.691949 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet fdbd697a90534ff3b7a5e17780207d88 (table rebalance_test_table_8 [id=87f74cad3e4e48a389dd92a283273233]) successfully deleted
I20240503 02:01:09.730079 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.730875 21721 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.731171 21721 tablet_bootstrap.cc:492] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:09.731323 21721 ts_tablet_manager.cc:1367] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.005s
I20240503 02:01:09.731740 21721 raft_consensus.cc:348] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:09.731897 21721 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:09.732031 21721 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:09.732808 21721 ts_tablet_manager.cc:1398] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 02:01:09.733469 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-b4b734308d3b42958e3a2830a83d44e2 received from {username='slave'} at 127.0.99.136:56441
I20240503 02:01:09.733554 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-b4b734308d3b42958e3a2830a83d44e2 on tablet b4b734308d3b42958e3a2830a83d44e2 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:09.735376 21759 ts_tablet_manager.cc:902] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:09.735844 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.736078 21759 tablet_copy_client.cc:308] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:09.736573 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 1c464b6e30bb414bbe70ed030c5f481c from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:54909)
I20240503 02:01:09.736690 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 1c464b6e30bb414bbe70ed030c5f481c from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:54909: session id = 9e39ab9b64544988b968015fe9fa06e6-1c464b6e30bb414bbe70ed030c5f481c
I20240503 02:01:09.737648 13055 tablet_copy_source_session.cc:205] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:09.740206 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c464b6e30bb414bbe70ed030c5f481c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:09.746776 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.752710 21759 tablet_copy_client.cc:783] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:09.753003 21759 tablet_copy_client.cc:647] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:09.758101 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.760990 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.758172 21759 tablet_copy_client.cc:523] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:09.778501 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:09.780057 21759 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:09.787928 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.814498 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.830519 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.836928 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.844727 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.860956 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.870950 12776 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:09.871341 21579 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.883790 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:09.887028 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.889097 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.893898 12886 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:09.895905 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:09.900923 12756 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.908358 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.910895 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.911491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:09.932493 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:09.933001 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.942814 21759 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:09.943140 21759 tablet_bootstrap.cc:492] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:09.943338 21759 ts_tablet_manager.cc:1367] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.163s user 0.025s sys 0.000s
I20240503 02:01:09.943930 21759 raft_consensus.cc:348] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.944072 21759 raft_consensus.cc:729] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:09.944214 21759 consensus_queue.cc:260] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:09.944538 21759 ts_tablet_manager.cc:1398] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:01:09.949754 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-1c464b6e30bb414bbe70ed030c5f481c received from {username='slave'} at 127.0.99.133:54909
I20240503 02:01:09.949842 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-1c464b6e30bb414bbe70ed030c5f481c on tablet 1c464b6e30bb414bbe70ed030c5f481c with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:09.954753 12905 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:09.966157 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:09.970841 21846 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:09.999042 14540 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:10.000842 14802 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:10.001595 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:10.001860 21850 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:10.002034 21849 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:10.004168 21850 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.005990 14802 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.006661 12776 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8292c06377f445849acdc22d08bd612d"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42368
I20240503 02:01:10.007076 12776 raft_consensus.cc:595] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:10.007062 13038 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.007251 12632 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.011062 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet f5767bdfcc6249578f7c3ada3c08da6e with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:10.011304 12776 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:10.011389 21732 tablet_replica.cc:330] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:10.011519 21732 raft_consensus.cc:2229] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.011618 21732 raft_consensus.cc:2258] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.012284 21732 ts_tablet_manager.cc:1875] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.012688 14934 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:10.014461 21732 ts_tablet_manager.cc:1888] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:01:10.014531 21732 log.cc:1196] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5767bdfcc6249578f7c3ada3c08da6e
I20240503 02:01:10.015125 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet f5767bdfcc6249578f7c3ada3c08da6e (table rebalance_test_table_2 [id=3f4d2c6e338a4a099ebc28f1418baf11]) successfully deleted
I20240503 02:01:10.015775 19794 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.021044 19794 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.023592 12631 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 5 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.026839 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:10.027334 12776 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
W20240503 02:01:10.028512 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1f9c8efd2040d6842c6b9f7c6173aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:10.031167 14933 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:10.031320 14934 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.034052 21925 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.034544 21925 raft_consensus.cc:979] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: : Instructing follower 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:10.034659 21925 raft_consensus.cc:1067] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Signalling peer 03ced8676740485e86ebe292f8ee4175 to start an election
I20240503 02:01:10.035291 13167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8292c06377f445849acdc22d08bd612d"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
from {username='slave'} at 127.0.99.129:38273
I20240503 02:01:10.035410 13167 raft_consensus.cc:482] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:10.035471 13167 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:10.036489 13167 raft_consensus.cc:504] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:10.036971 21931 tablet_replica.cc:330] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:10.037024 13167 leader_election.cc:288] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:01:10.037158 21931 raft_consensus.cc:2229] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.037274 21931 raft_consensus.cc:2258] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.037509 14665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d23425fa2b5848c8b1a2b737d75e69eb"
I20240503 02:01:10.037626 14665 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:10.037796 21931 ts_tablet_manager.cc:1875] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.037814 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:01:10.037910 12776 raft_consensus.cc:3032] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:10.037945 12776 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:01:10.038064 12776 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:10.038197 12776 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:10.038645 14665 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:10.039158 12776 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:10.039448 13103 leader_election.cc:302] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, bbba999bf95446a68b6586608b33ee16, d23425fa2b5848c8b1a2b737d75e69eb; no voters:
I20240503 02:01:10.039793 21261 raft_consensus.cc:2781] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:10.039899 21261 raft_consensus.cc:686] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Becoming Leader. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:10.040038 21261 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:10.040246 14934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8292c06377f445849acdc22d08bd612d" candidate_uuid: "03ced8676740485e86ebe292f8ee4175" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "558837c679f246dcb3241d66a37d70f1"
I20240503 02:01:10.040366 14934 raft_consensus.cc:3037] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:10.041041 21924 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:10.041319 14934 raft_consensus.cc:2454] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ced8676740485e86ebe292f8ee4175 in term 2.
I20240503 02:01:10.041380 14933 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:10.042649 12631 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 reported cstate change: term changed from 1 to 2, leader changed from bbba999bf95446a68b6586608b33ee16 (127.0.99.129) to 03ced8676740485e86ebe292f8ee4175 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.045771 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:10.046865 21931 ts_tablet_manager.cc:1888] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:01:10.046941 21931 log.cc:1196] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:01:10.047556 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet ae1f9c8efd2040d6842c6b9f7c6173aa (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:10.048038 12627 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:10.056078 14933 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:10.057814 14802 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:01:10.058254 21027 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:10.062716 14802 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.065222 12631 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.067613 21947 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.068128 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e314466c01488bb661c596c405db51 with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:10.068341 21931 tablet_replica.cc:330] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:10.068462 21931 raft_consensus.cc:2229] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.068594 21931 raft_consensus.cc:2258] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.069197 21931 ts_tablet_manager.cc:1875] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.071570 21931 ts_tablet_manager.cc:1888] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 02:01:10.071636 21931 log.cc:1196] T 86e314466c01488bb661c596c405db51 P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/86e314466c01488bb661c596c405db51
I20240503 02:01:10.072317 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 86e314466c01488bb661c596c405db51 (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:01:10.074410 14933 consensus_queue.cc:237] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:10.076189 14802 raft_consensus.cc:1261] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:01:10.078665 21947 consensus_queue.cc:1035] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:01:10.080987 21027 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:10.083412 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 86e314466c01488bb661c596c405db51 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:10.083943 12632 catalog_manager.cc:5484] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:10.083933 14802 raft_consensus.cc:2932] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
W20240503 02:01:10.085830 14870 consensus_peers.cc:485] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86e314466c01488bb661c596c405db51. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:10.129151 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.137530 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.144613 14934 raft_consensus.cc:1203] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:10.152266 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.156392 22051 ts_tablet_manager.cc:902] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:10.156903 22051 tablet_copy_client.cc:308] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:10.158151 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet fdbd697a90534ff3b7a5e17780207d88 from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:37179)
I20240503 02:01:10.158257 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet fdbd697a90534ff3b7a5e17780207d88 from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:37179: session id = d23425fa2b5848c8b1a2b737d75e69eb-fdbd697a90534ff3b7a5e17780207d88
I20240503 02:01:10.159360 13055 tablet_copy_source_session.cc:205] T fdbd697a90534ff3b7a5e17780207d88 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:10.162151 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.170157 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbd697a90534ff3b7a5e17780207d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:10.177450 22051 tablet_copy_client.cc:783] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:10.177742 22051 tablet_copy_client.cc:647] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:10.179386 22051 tablet_copy_client.cc:523] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:10.184789 22051 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:10.192946 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.201207 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.208642 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.215129 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.226223 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.229936 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.231932 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.232061 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.258631 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.258787 14540 raft_consensus.cc:1203] T 1c464b6e30bb414bbe70ed030c5f481c P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:10.263370 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.265349 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.266919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.285669 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.297519 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.297739 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.300036 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.303216 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.312286 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.323830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.324661 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.365391 22051 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:10.365875 22051 tablet_bootstrap.cc:492] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:10.366200 22051 ts_tablet_manager.cc:1367] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.181s user 0.024s sys 0.004s
I20240503 02:01:10.366760 22051 raft_consensus.cc:348] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:10.366994 22051 raft_consensus.cc:729] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:10.367175 22051 consensus_queue.cc:260] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:10.368247 22051 ts_tablet_manager.cc:1398] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:10.368682 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-fdbd697a90534ff3b7a5e17780207d88 received from {username='slave'} at 127.0.99.134:37179
I20240503 02:01:10.368753 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-fdbd697a90534ff3b7a5e17780207d88 on tablet fdbd697a90534ff3b7a5e17780207d88 with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:10.405474 21759 ts_tablet_manager.cc:902] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:10.405982 21759 tablet_copy_client.cc:308] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:10.406461 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:40631)
I20240503 02:01:10.406581 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:40631: session id = 9e39ab9b64544988b968015fe9fa06e6-ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:01:10.407866 12796 tablet_copy_source_session.cc:205] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:10.410038 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:10.413300 21759 tablet_copy_client.cc:783] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:10.413524 21759 tablet_copy_client.cc:647] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:10.414737 21759 tablet_copy_client.cc:523] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:10.416344 21759 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:10.444070 14665 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:10.445902 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 02:01:10.446439 21645 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:10.448241 21645 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.451098 12628 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 3 leader_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.451195 12905 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.455227 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a95f88ac1c459ba01be0743297f696 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:10.456983 13167 consensus_queue.cc:237] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:10.458357 14802 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:10.459265 21261 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:10.460647 12905 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:10.461127 19920 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:10.463613 22186 tablet_replica.cc:330] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:10.464166 22186 raft_consensus.cc:2229] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.464442 22186 raft_consensus.cc:2258] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.464687 21261 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.465764 22186 ts_tablet_manager.cc:1875] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.466936 12905 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.467375 12626 catalog_manager.cc:5484] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 4 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.471432 13016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8108933516894d4190ef9a0cf6063be8 with delete_type TABLET_DATA_TOMBSTONED (TS 22b582a5f4774861a5619c51198d39c5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55594
I20240503 02:01:10.471668 21931 tablet_replica.cc:330] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: stopping tablet replica
I20240503 02:01:10.471836 21931 raft_consensus.cc:2229] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.471957 21931 raft_consensus.cc:2258] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.472481 21931 ts_tablet_manager.cc:1875] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.472975 14802 raft_consensus.cc:2932] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.474669 21931 ts_tablet_manager.cc:1888] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:01:10.474735 21931 log.cc:1196] T 8108933516894d4190ef9a0cf6063be8 P 22b582a5f4774861a5619c51198d39c5: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8108933516894d4190ef9a0cf6063be8
I20240503 02:01:10.475673 12621 catalog_manager.cc:4830] TS 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): tablet 8108933516894d4190ef9a0cf6063be8 (table rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]) successfully deleted
I20240503 02:01:10.476536 22186 ts_tablet_manager.cc:1888] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240503 02:01:10.477665 22186 log.cc:1196] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/13a95f88ac1c459ba01be0743297f696
I20240503 02:01:10.478331 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 13a95f88ac1c459ba01be0743297f696 (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
I20240503 02:01:10.483187 14665 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:10.485533 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:01:10.486132 21645 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:10.487823 21645 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.489676 12905 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
W20240503 02:01:10.491499 14605 consensus_peers.cc:485] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb -> Peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171): Couldn't send request to peer 754fc04ea9914073a56c0cf164f63240. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13a95f88ac1c459ba01be0743297f696. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:10.491979 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 13a95f88ac1c459ba01be0743297f696 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:10.492560 12630 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 8 to 9, VOTER 754fc04ea9914073a56c0cf164f63240 (127.0.99.135) added. New cstate: current_term: 3 leader_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:10.518594 21759 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:10.519074 21759 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:10.519292 21759 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.103s user 0.036s sys 0.010s
I20240503 02:01:10.520087 21759 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:10.520246 21759 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:10.520437 21759 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:10.520915 21759 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:01:10.521520 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-ae1f9c8efd2040d6842c6b9f7c6173aa received from {username='slave'} at 127.0.99.133:40631
I20240503 02:01:10.521615 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-ae1f9c8efd2040d6842c6b9f7c6173aa on tablet ae1f9c8efd2040d6842c6b9f7c6173aa with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:10.529350 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.537988 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.544425 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.545010 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.547801 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.559861 21759 ts_tablet_manager.cc:902] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:01:10.561295 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.563864 21759 tablet_copy_client.cc:308] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41447
I20240503 02:01:10.574469 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.574980 14954 tablet_copy_service.cc:140] P 558837c679f246dcb3241d66a37d70f1: Received BeginTabletCopySession request for tablet 86e314466c01488bb661c596c405db51 from peer 9e39ab9b64544988b968015fe9fa06e6 ({username='slave'} at 127.0.99.133:52831)
I20240503 02:01:10.575093 14954 tablet_copy_service.cc:161] P 558837c679f246dcb3241d66a37d70f1: Beginning new tablet copy session on tablet 86e314466c01488bb661c596c405db51 from peer 9e39ab9b64544988b968015fe9fa06e6 at {username='slave'} at 127.0.99.133:52831: session id = 9e39ab9b64544988b968015fe9fa06e6-86e314466c01488bb661c596c405db51
I20240503 02:01:10.576074 14954 tablet_copy_source_session.cc:205] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:10.576925 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e314466c01488bb661c596c405db51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:10.579018 21759 tablet_copy_client.cc:783] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:10.579164 21759 tablet_copy_client.cc:647] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:10.579636 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.580503 21759 tablet_copy_client.cc:523] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:10.581686 21759 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap starting.
I20240503 02:01:10.590488 12776 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:10.591053 22202 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.600769 14665 raft_consensus.cc:1203] T fdbd697a90534ff3b7a5e17780207d88 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:10.604276 14934 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:10.606374 22202 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.657984 14665 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:10.658360 22202 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.728847 21759 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:10.729131 21759 tablet_bootstrap.cc:492] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Bootstrap complete.
I20240503 02:01:10.729303 21759 ts_tablet_manager.cc:1367] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Time spent bootstrapping tablet: real 0.148s user 0.023s sys 0.004s
I20240503 02:01:10.729699 21759 raft_consensus.cc:348] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:10.729800 21759 raft_consensus.cc:729] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Initialized, Role: FOLLOWER
I20240503 02:01:10.729965 21759 consensus_queue.cc:260] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:10.730325 21759 ts_tablet_manager.cc:1398] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:01:10.731125 14954 tablet_copy_service.cc:342] P 558837c679f246dcb3241d66a37d70f1: Request end of tablet copy session 9e39ab9b64544988b968015fe9fa06e6-86e314466c01488bb661c596c405db51 received from {username='slave'} at 127.0.99.133:52831
I20240503 02:01:10.731201 14954 tablet_copy_service.cc:433] P 558837c679f246dcb3241d66a37d70f1: ending tablet copy session 9e39ab9b64544988b968015fe9fa06e6-86e314466c01488bb661c596c405db51 on tablet 86e314466c01488bb661c596c405db51 with peer 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:10.738879 12905 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:10.740123 14665 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:10.741408 21862 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:10.754382 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.773896 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.787503 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.788441 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.791162 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.792944 21576 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.793433 14665 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.794389 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.796572 12634 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:10.801674 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 490776bd408a48a9bcbbc7a501baa8ce with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:10.802057 22186 tablet_replica.cc:330] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:10.802199 22186 raft_consensus.cc:2229] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:10.802364 22186 raft_consensus.cc:2258] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:10.802649 12905 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:10.802966 22186 ts_tablet_manager.cc:1875] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:10.803786 14665 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:10.804291 21862 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:10.805217 22186 ts_tablet_manager.cc:1888] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:10.805282 22186 log.cc:1196] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/490776bd408a48a9bcbbc7a501baa8ce
W20240503 02:01:10.805531 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 490776bd408a48a9bcbbc7a501baa8ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:10.806088 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 490776bd408a48a9bcbbc7a501baa8ce (table rebalance_test_table_4 [id=99b49350f69a4139a63c1dd150969f7f]) successfully deleted
I20240503 02:01:10.806783 21862 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.808295 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 490776bd408a48a9bcbbc7a501baa8ce with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:10.808498 14665 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.810333 12630 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:10.820724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.838394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.907747 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.909428 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.918099 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.918880 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:10.920764 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.930841 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.933971 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.939492 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.940191 14540 raft_consensus.cc:1203] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:01:10.955102 14540 raft_consensus.cc:1203] T 86e314466c01488bb661c596c405db51 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 02:01:10.956116 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.963632 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.966181 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.972296 14914 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:10.980701 13149 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:01:10.991545 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:10.992671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:10.993495 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.000156 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.007133 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.020138 22479 ts_tablet_manager.cc:902] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Initiating tablet copy from peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:01:11.020725 22479 tablet_copy_client.cc:308] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40697
I20240503 02:01:11.022027 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.023448 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.033003 14691 tablet_copy_service.cc:140] P d23425fa2b5848c8b1a2b737d75e69eb: Received BeginTabletCopySession request for tablet 13a95f88ac1c459ba01be0743297f696 from peer 754fc04ea9914073a56c0cf164f63240 ({username='slave'} at 127.0.99.135:59225)
I20240503 02:01:11.033123 14691 tablet_copy_service.cc:161] P d23425fa2b5848c8b1a2b737d75e69eb: Beginning new tablet copy session on tablet 13a95f88ac1c459ba01be0743297f696 from peer 754fc04ea9914073a56c0cf164f63240 at {username='slave'} at 127.0.99.135:59225: session id = 754fc04ea9914073a56c0cf164f63240-13a95f88ac1c459ba01be0743297f696
I20240503 02:01:11.033802 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.034461 14691 tablet_copy_source_session.cc:205] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:11.035650 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:11.038748 22479 tablet_copy_client.cc:783] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:11.038918 22479 tablet_copy_client.cc:647] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:11.044059 22479 tablet_copy_client.cc:523] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:11.045576 22479 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Bootstrap starting.
I20240503 02:01:11.047856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.062647 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.067817 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.104113 13167 consensus_queue.cc:237] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:11.105994 14665 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:11.106032 13167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55106
I20240503 02:01:11.106127 13167 raft_consensus.cc:595] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:11.106410 22202 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:11.111891 13167 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8"
dest_uuid: "03ced8676740485e86ebe292f8ee4175"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55122
I20240503 02:01:11.112031 13167 raft_consensus.cc:595] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:01:11.113512 19920 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.118273 14665 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.119382 12626 catalog_manager.cc:5484] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.122604 22479 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:11.122948 22479 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Bootstrap complete.
I20240503 02:01:11.123131 22479 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Time spent bootstrapping tablet: real 0.078s user 0.032s sys 0.000s
I20240503 02:01:11.123227 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet e6cc67c73560449e81303a948df8f9df with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:11.123603 22479 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.123723 22479 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Initialized, Role: FOLLOWER
I20240503 02:01:11.123867 22479 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.124622 22479 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:11.124591 13167 consensus_queue.cc:237] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:01:11.125026 21261 raft_consensus.cc:979] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175: : Instructing follower 558837c679f246dcb3241d66a37d70f1 to start an election
I20240503 02:01:11.125278 14691 tablet_copy_service.cc:342] P d23425fa2b5848c8b1a2b737d75e69eb: Request end of tablet copy session 754fc04ea9914073a56c0cf164f63240-13a95f88ac1c459ba01be0743297f696 received from {username='slave'} at 127.0.99.135:59225
I20240503 02:01:11.125366 14691 tablet_copy_service.cc:433] P d23425fa2b5848c8b1a2b737d75e69eb: ending tablet copy session 754fc04ea9914073a56c0cf164f63240-13a95f88ac1c459ba01be0743297f696 on tablet 13a95f88ac1c459ba01be0743297f696 with peer 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:11.125437 21261 raft_consensus.cc:1067] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Signalling peer 558837c679f246dcb3241d66a37d70f1 to start an election
I20240503 02:01:11.126677 14665 raft_consensus.cc:1261] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:01:11.126741 13103 consensus_peers.cc:485] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 -> Peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): Couldn't send request to peer bbba999bf95446a68b6586608b33ee16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6cc67c73560449e81303a948df8f9df. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.126725 14934 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2"
dest_uuid: "558837c679f246dcb3241d66a37d70f1"
from {username='slave'} at 127.0.99.132:51879
I20240503 02:01:11.126853 14934 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:11.126925 14934 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.127408 22541 consensus_queue.cc:1035] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:11.128034 14934 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.128409 14934 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [CANDIDATE]: Term 2 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:01:11.129348 19920 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:01:11.129801 14540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "558837c679f246dcb3241d66a37d70f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "9e39ab9b64544988b968015fe9fa06e6"
I20240503 02:01:11.129922 14540 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.130086 14665 raft_consensus.cc:2932] T e6cc67c73560449e81303a948df8f9df P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:01:11.130950 14540 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558837c679f246dcb3241d66a37d70f1 in term 2.
I20240503 02:01:11.130959 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e6cc67c73560449e81303a948df8f9df with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:11.131356 14870 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6; no voters:
I20240503 02:01:11.131518 21947 raft_consensus.cc:2781] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:11.131639 21947 raft_consensus.cc:686] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [term 2 LEADER]: Becoming Leader. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Running, Role: LEADER
I20240503 02:01:11.131786 21947 consensus_queue.cc:237] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.132184 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b734308d3b42958e3a2830a83d44e2" candidate_uuid: "558837c679f246dcb3241d66a37d70f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:01:11.132273 13167 raft_consensus.cc:3032] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:11.132309 13167 raft_consensus.cc:729] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:11.132444 13167 consensus_queue.cc:260] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.132583 13167 raft_consensus.cc:3037] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.133452 13167 raft_consensus.cc:2454] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558837c679f246dcb3241d66a37d70f1 in term 2.
I20240503 02:01:11.134079 12626 catalog_manager.cc:5484] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:01:11.134904 22536 tablet_replica.cc:330] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:11.134977 12630 catalog_manager.cc:5484] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: term changed from 1 to 2, leader changed from 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) to 558837c679f246dcb3241d66a37d70f1 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "558837c679f246dcb3241d66a37d70f1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:11.135054 22536 raft_consensus.cc:2229] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:11.135222 22536 raft_consensus.cc:2258] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:11.135735 22536 ts_tablet_manager.cc:1875] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:11.138916 22536 ts_tablet_manager.cc:1888] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:11.138993 22536 log.cc:1196] T e6cc67c73560449e81303a948df8f9df P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e6cc67c73560449e81303a948df8f9df
I20240503 02:01:11.143303 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet e6cc67c73560449e81303a948df8f9df (table rebalance_test_table_1 [id=850edfd853f047b19d57394a1b67d1f2]) successfully deleted
I20240503 02:01:11.164363 13038 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:11.165768 14665 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:11.166229 22302 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:11.185159 22302 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.189145 14665 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.191941 12626 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.196993 13038 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.197629 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 613074d3c4514bd5b64a06c59af09a56 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:11.198148 22186 tablet_replica.cc:330] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:11.198297 22186 raft_consensus.cc:2229] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:11.198448 22186 raft_consensus.cc:2258] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:11.198690 14665 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:11.199043 22186 ts_tablet_manager.cc:1875] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:11.199060 22302 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:01:11.199277 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 613074d3c4514bd5b64a06c59af09a56. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.201187 22302 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:11.202025 22186 ts_tablet_manager.cc:1888] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:11.202090 22186 log.cc:1196] T 613074d3c4514bd5b64a06c59af09a56 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/613074d3c4514bd5b64a06c59af09a56
I20240503 02:01:11.202579 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 613074d3c4514bd5b64a06c59af09a56 (table rebalance_test_table_5 [id=baadc882e6ce4f52b3d62a2bdf4a4b63]) successfully deleted
I20240503 02:01:11.203269 14665 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:11.205432 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 613074d3c4514bd5b64a06c59af09a56 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:11.205654 12626 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:11.245146 22664 ts_tablet_manager.cc:902] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:01:11.245648 22664 tablet_copy_client.cc:308] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:01:11.245674 13149 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:01:11.246116 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 490776bd408a48a9bcbbc7a501baa8ce from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:47631)
I20240503 02:01:11.246219 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 490776bd408a48a9bcbbc7a501baa8ce from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:47631: session id = 558837c679f246dcb3241d66a37d70f1-490776bd408a48a9bcbbc7a501baa8ce
I20240503 02:01:11.247185 12927 tablet_copy_source_session.cc:205] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:11.253096 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.260819 22664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:11.263927 22664 tablet_copy_client.cc:783] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:11.264093 22664 tablet_copy_client.cc:647] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:11.270519 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.271875 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.272713 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.276856 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.281266 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.281780 12887 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.282728 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.288573 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.299794 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.303192 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.308185 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.315985 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.356806 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.372464 22541 raft_consensus.cc:979] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175: : Instructing follower 96b8f7d7c1ce4e2cad5af6caabfd8c4a to start an election
I20240503 02:01:11.372565 22541 raft_consensus.cc:1067] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Signalling peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a to start an election
I20240503 02:01:11.373034 12907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8"
dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
from {username='slave'} at 127.0.99.132:48855
I20240503 02:01:11.373164 12907 raft_consensus.cc:482] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:11.373212 12907 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:11.374208 12907 raft_consensus.cc:504] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.374830 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:01:11.374946 13167 raft_consensus.cc:3032] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:01:11.374979 13167 raft_consensus.cc:729] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Running, Role: LEADER
I20240503 02:01:11.375097 13167 consensus_queue.cc:260] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.375217 13167 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:11.376148 13167 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 3.
I20240503 02:01:11.376425 12842 leader_election.cc:302] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
I20240503 02:01:11.376585 21576 raft_consensus.cc:2781] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:11.376765 21576 raft_consensus.cc:686] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 LEADER]: Becoming Leader. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Running, Role: LEADER
I20240503 02:01:11.376916 21576 consensus_queue.cc:237] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.378409 12907 leader_election.cc:288] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 3 election: Requested vote from peers 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157), 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:01:11.378471 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8108933516894d4190ef9a0cf6063be8" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "754fc04ea9914073a56c0cf164f63240"
I20240503 02:01:11.378602 14802 raft_consensus.cc:3037] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:11.378738 12626 catalog_manager.cc:5484] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: term changed from 2 to 3, leader changed from 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) to 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130). New cstate: current_term: 3 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:11.379621 14802 raft_consensus.cc:2454] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b8f7d7c1ce4e2cad5af6caabfd8c4a in term 3.
I20240503 02:01:11.388556 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.393067 22664 tablet_copy_client.cc:523] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:11.394515 22664 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:11.438760 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.441707 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.444185 14802 raft_consensus.cc:1203] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240503 02:01:11.449539 22664 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:11.449877 22664 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:11.450088 22664 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.056s user 0.026s sys 0.000s
I20240503 02:01:11.450594 22664 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.450723 22664 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:11.450865 22664 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.450909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.451617 22664 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:01:11.452075 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-490776bd408a48a9bcbbc7a501baa8ce received from {username='slave'} at 127.0.99.136:47631
I20240503 02:01:11.452164 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-490776bd408a48a9bcbbc7a501baa8ce on tablet 490776bd408a48a9bcbbc7a501baa8ce with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:11.459407 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.461606 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.470373 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.475978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.506314 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.553710 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.555048 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.560969 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.565739 14519 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.566987 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.578125 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.584065 22811 ts_tablet_manager.cc:902] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:01:11.584770 22811 tablet_copy_client.cc:308] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:01:11.585343 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet e6cc67c73560449e81303a948df8f9df from peer bbba999bf95446a68b6586608b33ee16 ({username='slave'} at 127.0.99.129:38273)
I20240503 02:01:11.585458 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet e6cc67c73560449e81303a948df8f9df from peer bbba999bf95446a68b6586608b33ee16 at {username='slave'} at 127.0.99.129:38273: session id = bbba999bf95446a68b6586608b33ee16-e6cc67c73560449e81303a948df8f9df
I20240503 02:01:11.586503 13189 tablet_copy_source_session.cc:205] T e6cc67c73560449e81303a948df8f9df P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:11.587838 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cc67c73560449e81303a948df8f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:11.590389 22811 tablet_copy_client.cc:783] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:11.590565 22811 tablet_copy_client.cc:647] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:11.597694 22811 tablet_copy_client.cc:523] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:11.600188 22811 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:01:11.609535 13038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4"
dest_uuid: "22b582a5f4774861a5619c51198d39c5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46790
I20240503 02:01:11.609676 13038 raft_consensus.cc:595] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:01:11.612671 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.626662 14539 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:11.627208 22547 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:11.634898 22623 raft_consensus.cc:979] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: : Instructing follower 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:11.635006 22623 raft_consensus.cc:1067] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Signalling peer 9e39ab9b64544988b968015fe9fa06e6 to start an election
I20240503 02:01:11.635396 14539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4"
dest_uuid: "9e39ab9b64544988b968015fe9fa06e6"
from {username='slave'} at 127.0.99.131:56109
I20240503 02:01:11.635424 13167 raft_consensus.cc:1261] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Refusing update from remote peer 558837c679f246dcb3241d66a37d70f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:11.635514 14539 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:01:11.635572 14539 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.635934 22547 consensus_queue.cc:1035] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:11.636716 14539 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:11.637104 14539 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:11.637670 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:11.637794 13038 raft_consensus.cc:3032] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:01:11.637835 13038 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:01:11.637986 13038 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:11.638152 13038 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.638233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.638713 22811 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:11.639073 22811 tablet_bootstrap.cc:492] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Bootstrap complete.
I20240503 02:01:11.639039 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "9e39ab9b64544988b968015fe9fa06e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:11.639156 12907 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:01:11.639200 13038 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:11.639254 22811 ts_tablet_manager.cc:1367] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.000s
I20240503 02:01:11.639669 22811 raft_consensus.cc:348] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:01:11.640033 14476 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 9e39ab9b64544988b968015fe9fa06e6; no voters:
I20240503 02:01:11.640199 21849 raft_consensus.cc:2781] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:01:11.640192 12907 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e39ab9b64544988b968015fe9fa06e6 in term 2.
I20240503 02:01:11.640341 21849 raft_consensus.cc:686] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Becoming Leader. State: Replica: 9e39ab9b64544988b968015fe9fa06e6, State: Running, Role: LEADER
I20240503 02:01:11.640455 21849 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:11.639799 22811 raft_consensus.cc:729] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:01:11.642011 22811 consensus_queue.cc:260] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:01:11.642038 12626 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: term changed from 1 to 2, leader changed from 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) to 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:11.643033 22811 ts_tablet_manager.cc:1398] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:01:11.646718 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session bbba999bf95446a68b6586608b33ee16-e6cc67c73560449e81303a948df8f9df received from {username='slave'} at 127.0.99.129:38273
I20240503 02:01:11.646811 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session bbba999bf95446a68b6586608b33ee16-e6cc67c73560449e81303a948df8f9df on tablet e6cc67c73560449e81303a948df8f9df with peer bbba999bf95446a68b6586608b33ee16
I20240503 02:01:11.664033 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.664664 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.667097 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:11.668169 22664 ts_tablet_manager.cc:902] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:11.668643 22664 tablet_copy_client.cc:308] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:11.672308 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 613074d3c4514bd5b64a06c59af09a56 from peer 558837c679f246dcb3241d66a37d70f1 ({username='slave'} at 127.0.99.136:51355)
I20240503 02:01:11.672421 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 613074d3c4514bd5b64a06c59af09a56 from peer 558837c679f246dcb3241d66a37d70f1 at {username='slave'} at 127.0.99.136:51355: session id = 558837c679f246dcb3241d66a37d70f1-613074d3c4514bd5b64a06c59af09a56
I20240503 02:01:11.673321 13055 tablet_copy_source_session.cc:205] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:11.676405 22664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:11.678756 22664 tablet_copy_client.cc:783] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:11.678926 22664 tablet_copy_client.cc:647] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:11.679706 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.693087 14934 raft_consensus.cc:1203] T 490776bd408a48a9bcbbc7a501baa8ce P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:11.694427 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.704203 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.718086 22664 tablet_copy_client.cc:523] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:11.719921 22664 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Bootstrap starting.
I20240503 02:01:11.725893 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.736434 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.788355 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.796087 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.802848 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.803742 13167 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:01:11.804252 22698 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:11.808617 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.817826 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:11.840358 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.841809 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:11.842619 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.845644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.853997 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.860656 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:11.862985 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.889536 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.895237 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:11.902313 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:11.917833 14802 raft_consensus.cc:1261] T 8108933516894d4190ef9a0cf6063be8 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:01:11.921597 22698 consensus_queue.cc:1035] T 8108933516894d4190ef9a0cf6063be8 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:11.926048 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:11.937008 22664 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:11.937389 22664 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Bootstrap complete.
I20240503 02:01:11.937575 22664 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Time spent bootstrapping tablet: real 0.218s user 0.026s sys 0.000s
I20240503 02:01:11.938418 22664 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.938539 22664 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558837c679f246dcb3241d66a37d70f1, State: Initialized, Role: FOLLOWER
I20240503 02:01:11.938678 22664 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:11.939065 22664 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:11.939790 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 558837c679f246dcb3241d66a37d70f1-613074d3c4514bd5b64a06c59af09a56 received from {username='slave'} at 127.0.99.136:51355
I20240503 02:01:11.939882 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 558837c679f246dcb3241d66a37d70f1-613074d3c4514bd5b64a06c59af09a56 on tablet 613074d3c4514bd5b64a06c59af09a56 with peer 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:11.944232 14665 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
W20240503 02:01:11.946283 14606 consensus_peers.cc:485] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.946709 12907 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:01:11.947777 22184 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:01:11.947912 14802 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:01:11.948477 22184 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:01:11.956851 22184 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:11.960206 14802 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:11.959841 12907 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:11.962376 12629 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 3 leader_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:11.964954 14802 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:11.966840 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:11.967567 22965 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:11.969847 21468 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:11.970414 12907 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:11.972394 12634 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 5 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:11.976760 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 007feed148fb44e9a6fe9d449b6d8f9d with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:11.977401 23010 tablet_replica.cc:330] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:11.977587 23010 raft_consensus.cc:2229] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:11.977730 23010 raft_consensus.cc:2258] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:11.978276 23010 ts_tablet_manager.cc:1875] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:11.980507 14802 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:11.982225 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 02:01:11.983184 14737 consensus_peers.cc:485] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007feed148fb44e9a6fe9d449b6d8f9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:11.983422 22965 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:11.988440 21468 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.989636 12907 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.991541 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 007feed148fb44e9a6fe9d449b6d8f9d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:11.992177 23010 ts_tablet_manager.cc:1888] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:01:11.992377 23010 log.cc:1196] T 007feed148fb44e9a6fe9d449b6d8f9d P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/007feed148fb44e9a6fe9d449b6d8f9d
I20240503 02:01:11.992269 12626 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:11.998137 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 007feed148fb44e9a6fe9d449b6d8f9d (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
I20240503 02:01:12.006456 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.006744 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.016443 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.019912 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.039938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.044687 13038 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:12.048913 14934 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:12.049515 12776 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:12.049635 22622 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:12.049937 22623 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:01:12.051748 12974 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86b733f38fd4f08bd40865b99e2e7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.056445 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.056443 22623 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.057986 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.059572 12627 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.056545 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:12.062415 14934 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.064124 14933 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:01:12.065294 22622 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:12.065912 12776 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.079344 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.080950 22623 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:12.088827 12626 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:12.093349 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:01:12.095268 14933 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:12.097103 14934 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:12.098987 23071 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:12.102593 13167 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:01:12.102836 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet a105660c12094c71b3a4b80e661d9308 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:12.102773 14934 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:12.103132 23092 tablet_replica.cc:330] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:12.103297 23092 raft_consensus.cc:2229] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:12.103487 23092 raft_consensus.cc:2258] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:12.104076 23092 ts_tablet_manager.cc:1875] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:12.105335 12626 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:12.106086 22622 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:01:12.107306 23092 ts_tablet_manager.cc:1888] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:01:12.107364 23092 log.cc:1196] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a105660c12094c71b3a4b80e661d9308
I20240503 02:01:12.108356 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet a105660c12094c71b3a4b80e661d9308 (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:12.108800 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a105660c12094c71b3a4b80e661d9308 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:12.109953 22623 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:01:12.111173 12973 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.111493 12907 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:12.111968 22866 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:12.116250 14934 raft_consensus.cc:1203] T 613074d3c4514bd5b64a06c59af09a56 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:12.129526 13038 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:01:12.130066 22866 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:01:12.134773 12776 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:01:12.136945 14539 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 02:01:12.136957 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679): Couldn't send request to peer 22b582a5f4774861a5619c51198d39c5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.137295 14934 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:01:12.138064 21925 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:12.139125 23120 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:12.142944 21925 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:01:12.145418 14539 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 9e39ab9b64544988b968015fe9fa06e6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:01:12.146229 14934 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:01:12.147305 12630 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 8 to 9, VOTER 22b582a5f4774861a5619c51198d39c5 (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:01:12.166129 12776 raft_consensus.cc:1203] T e6cc67c73560449e81303a948df8f9df P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:01:12.216295 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
W20240503 02:01:12.218819 12973 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a105660c12094c71b3a4b80e661d9308. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.219290 14934 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:01:12.219731 22622 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:01:12.219929 12973 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.233743 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.240255 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.241802 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.245492 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.249190 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.251168 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.257458 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.259357 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.340827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.343381 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.345239 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.361588 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.382445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.402925 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.410125 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.413522 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.419974 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.431596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.432420 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.436348 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.440939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.446521 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.457226 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.462348 14539 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:12.463824 12774 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:01:12.464274 22866 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:12.467219 23337 ts_tablet_manager.cc:902] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:01:12.468066 23337 tablet_copy_client.cc:308] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40697
I20240503 02:01:12.468638 14691 tablet_copy_service.cc:140] P d23425fa2b5848c8b1a2b737d75e69eb: Received BeginTabletCopySession request for tablet 13a95f88ac1c459ba01be0743297f696 from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:58521)
I20240503 02:01:12.468736 14691 tablet_copy_service.cc:161] P d23425fa2b5848c8b1a2b737d75e69eb: Beginning new tablet copy session on tablet 13a95f88ac1c459ba01be0743297f696 from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:58521: session id = 03ced8676740485e86ebe292f8ee4175-13a95f88ac1c459ba01be0743297f696
I20240503 02:01:12.469714 14691 tablet_copy_source_session.cc:205] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.480171 23337 ts_tablet_manager.cc:1875] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:01:12.483481 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.488015 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.491676 23337 ts_tablet_manager.cc:1888] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.7
I20240503 02:01:12.494153 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a95f88ac1c459ba01be0743297f696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.496068 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.500591 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.506412 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.509539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.518162 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.522743 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.523909 22866 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.527666 12774 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.529831 12626 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: config changed from index 5 to 7, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
I20240503 02:01:12.538369 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 662782d61c6f472baab4d613b7e6f5cd with delete_type TABLET_DATA_TOMBSTONED (TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34680
I20240503 02:01:12.543186 14539 consensus_queue.cc:237] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:12.545094 12774 raft_consensus.cc:1261] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Refusing update from remote peer 9e39ab9b64544988b968015fe9fa06e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:12.545517 22866 consensus_queue.cc:1035] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:01:12.550943 14475 consensus_peers.cc:485] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:12.553941 22866 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:12.555967 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 662782d61c6f472baab4d613b7e6f5cd with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:01:12.556450 23337 tablet_copy_client.cc:783] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.556730 23337 tablet_copy_client.cc:647] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.557857 23347 ts_tablet_manager.cc:902] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:01:12.558115 12774 raft_consensus.cc:2932] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:12.558877 23347 tablet_copy_client.cc:308] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37171
I20240503 02:01:12.559101 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.559549 12626 catalog_manager.cc:5484] T 662782d61c6f472baab4d613b7e6f5cd P 9e39ab9b64544988b968015fe9fa06e6 reported cstate change: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "9e39ab9b64544988b968015fe9fa06e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:12.560588 14822 tablet_copy_service.cc:140] P 754fc04ea9914073a56c0cf164f63240: Received BeginTabletCopySession request for tablet 007feed148fb44e9a6fe9d449b6d8f9d from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:44693)
I20240503 02:01:12.560729 14822 tablet_copy_service.cc:161] P 754fc04ea9914073a56c0cf164f63240: Beginning new tablet copy session on tablet 007feed148fb44e9a6fe9d449b6d8f9d from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:44693: session id = d23425fa2b5848c8b1a2b737d75e69eb-007feed148fb44e9a6fe9d449b6d8f9d
I20240503 02:01:12.562079 14822 tablet_copy_source_session.cc:205] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.563268 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007feed148fb44e9a6fe9d449b6d8f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.565801 23347 tablet_copy_client.cc:783] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.565887 23354 tablet_replica.cc:330] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: stopping tablet replica
I20240503 02:01:12.566008 23347 tablet_copy_client.cc:647] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.566256 23354 raft_consensus.cc:2229] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:12.566380 23354 raft_consensus.cc:2258] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:12.566874 23354 ts_tablet_manager.cc:1875] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:12.568400 23337 tablet_copy_client.cc:523] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:12.570546 23337 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:01:12.585880 23357 ts_tablet_manager.cc:902] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:01:12.586766 23357 tablet_copy_client.cc:308] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:01:12.588008 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 22b582a5f4774861a5619c51198d39c5 ({username='slave'} at 127.0.99.131:50139)
I20240503 02:01:12.588124 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 22b582a5f4774861a5619c51198d39c5 at {username='slave'} at 127.0.99.131:50139: session id = 22b582a5f4774861a5619c51198d39c5-ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:01:12.588929 23354 ts_tablet_manager.cc:1888] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:01:12.589008 23354 log.cc:1196] T 662782d61c6f472baab4d613b7e6f5cd P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/662782d61c6f472baab4d613b7e6f5cd
I20240503 02:01:12.593511 12796 tablet_copy_source_session.cc:205] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.596621 23357 ts_tablet_manager.cc:1875] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:01:12.596792 23347 tablet_copy_client.cc:523] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:12.598484 23347 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:01:12.600625 23357 ts_tablet_manager.cc:1888] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 02:01:12.600874 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.602710 12620 catalog_manager.cc:4830] TS 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): tablet 662782d61c6f472baab4d613b7e6f5cd (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:12.603853 23357 tablet_copy_client.cc:783] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.604082 23357 tablet_copy_client.cc:647] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.605729 23357 tablet_copy_client.cc:523] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:12.607409 23357 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Bootstrap starting.
I20240503 02:01:12.621224 23337 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:12.621961 23337 tablet_bootstrap.cc:492] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:01:12.622175 23337 ts_tablet_manager.cc:1367] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.052s user 0.032s sys 0.000s
I20240503 02:01:12.622694 23337 raft_consensus.cc:348] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:01:12.622821 23337 raft_consensus.cc:729] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:01:12.623026 23337 consensus_queue.cc:260] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:01:12.623308 23337 ts_tablet_manager.cc:1398] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:01:12.623888 14691 tablet_copy_service.cc:342] P d23425fa2b5848c8b1a2b737d75e69eb: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-13a95f88ac1c459ba01be0743297f696 received from {username='slave'} at 127.0.99.132:58521
I20240503 02:01:12.623982 14691 tablet_copy_service.cc:433] P d23425fa2b5848c8b1a2b737d75e69eb: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-13a95f88ac1c459ba01be0743297f696 on tablet 13a95f88ac1c459ba01be0743297f696 with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:01:12.634217 23347 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:12.634541 23347 tablet_bootstrap.cc:492] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:12.634744 23347 ts_tablet_manager.cc:1367] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.000s
I20240503 02:01:12.635396 23347 raft_consensus.cc:348] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:12.635558 23347 raft_consensus.cc:729] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:12.635742 23347 consensus_queue.cc:260] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:12.636214 23347 ts_tablet_manager.cc:1398] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:01:12.638190 14822 tablet_copy_service.cc:342] P 754fc04ea9914073a56c0cf164f63240: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-007feed148fb44e9a6fe9d449b6d8f9d received from {username='slave'} at 127.0.99.134:44693
I20240503 02:01:12.638295 14822 tablet_copy_service.cc:433] P 754fc04ea9914073a56c0cf164f63240: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-007feed148fb44e9a6fe9d449b6d8f9d on tablet 007feed148fb44e9a6fe9d449b6d8f9d with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:12.651386 23357 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:01:12.651727 23357 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Bootstrap complete.
I20240503 02:01:12.651957 23357 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.000s
I20240503 02:01:12.652532 23357 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:01:12.652666 23357 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Initialized, Role: FOLLOWER
I20240503 02:01:12.652802 23357 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:01:12.654470 23357 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:01:12.655813 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 22b582a5f4774861a5619c51198d39c5-ae1f9c8efd2040d6842c6b9f7c6173aa received from {username='slave'} at 127.0.99.131:50139
I20240503 02:01:12.655896 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 22b582a5f4774861a5619c51198d39c5-ae1f9c8efd2040d6842c6b9f7c6173aa on tablet ae1f9c8efd2040d6842c6b9f7c6173aa with peer 22b582a5f4774861a5619c51198d39c5
I20240503 02:01:12.668057 23435 ts_tablet_manager.cc:902] T d86b733f38fd4f08bd40865b99e2e7b2 P 9e39ab9b64544988b968015fe9fa06e6: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
F20240503 02:01:12.668254 23435 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.queue_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.queue_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_cpu_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_cpu_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_wall_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_wall_time_us) (tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.queue_time_us, tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.queue_time_us) (tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.run_cpu_time_us, tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.run_cpu_time_us) (tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.run_wall_time_us, tablet-download-pool-0613ddd56ed346b3b6dfa9e7c3a4335e.run_wall_time_us) (tablet-download-pool-0633596226954c439d2f671c25eebfe6.queue_time_us, tablet-download-pool-0633596226954c439d2f671c25eebfe6.queue_time_us) (tablet-download-pool-0633596226954c439d2f671c25eebfe6.run_cpu_time_us, tablet-download-pool-0633596226954c439d2f671c25eebfe6.run_cpu_time_us) (tablet-download-pool-0633596226954c439d2f671c25eebfe6.run_wall_time_us, tablet-download-pool-0633596226954c439d2f671c25eebfe6.run_wall_time_us) (tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.queue_time_us, tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.queue_time_us) (tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.run_cpu_time_us, tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.run_cpu_time_us) (tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.run_wall_time_us, tablet-download-pool-063a7f2ed28a455697b000f52a4343a3.run_wall_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.queue_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.queue_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_cpu_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_cpu_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_wall_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_wall_time_us) (tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.queue_time_us, tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.queue_time_us) (tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.run_cpu_time_us, tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.run_cpu_time_us) (tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.run_wall_time_us, tablet-download-pool-141097686fe24c8f9956dcc1ad1d253c.run_wall_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.queue_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.queue_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_cpu_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_cpu_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_wall_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_wall_time_us) (tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.queue_time_us, tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.queue_time_us) (tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.run_cpu_time_us, tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.run_cpu_time_us) (tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.run_wall_time_us, tablet-download-pool-34e88e69beb64036bb676b771dabd2a4.run_wall_time_us) (tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.queue_time_us, tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.queue_time_us) (tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.run_cpu_time_us, tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.run_cpu_time_us) (tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.run_wall_time_us, tablet-download-pool-662782d61c6f472baab4d613b7e6f5cd.run_wall_time_us) (tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.queue_time_us, tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.queue_time_us) (tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.run_cpu_time_us, tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.run_cpu_time_us) (tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.run_wall_time_us, tablet-download-pool-6ad545d33bbc40458c75ed7c90d92ba7.run_wall_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.queue_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.queue_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.run_cpu_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.run_cpu_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.run_wall_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.run_wall_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.queue_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.queue_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_cpu_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_cpu_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_wall_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_wall_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.queue_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.queue_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_cpu_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_cpu_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_wall_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_wall_time_us) (tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.queue_time_us, tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.queue_time_us) (tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.queue_time_us, tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.queue_time_us) (tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.run_cpu_time_us, tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.run_cpu_time_us) (tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.run_wall_time_us, tablet-download-pool-f5767bdfcc6249578f7c3ada3c08da6e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f6ee24ecdcd google::LogMessage::Fail() at ??:0
@ 0x7f6ee24f0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6ee24ec7cc google::LogMessage::Flush() at ??:0
@ 0x7f6ee24edf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6ee3956044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6ee393af98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6ee3938efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f6ee64eb77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f6ee64f4a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:01:12.714927 23347 ts_tablet_manager.cc:902] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:12.715459 23347 tablet_copy_client.cc:308] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:12.716037 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet a105660c12094c71b3a4b80e661d9308 from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:37179)
I20240503 02:01:12.716158 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet a105660c12094c71b3a4b80e661d9308 from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:37179: session id = d23425fa2b5848c8b1a2b737d75e69eb-a105660c12094c71b3a4b80e661d9308
@ 0x7f6ee657576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:01:12.717468 13055 tablet_copy_source_session.cc:205] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.718773 23347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.721424 23347 tablet_copy_client.cc:783] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.721577 23347 tablet_copy_client.cc:647] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.722818 23347 tablet_copy_client.cc:523] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f6ee65894f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:01:12.724315 23347 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
@ 0x7f6ee6574226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f6ee6583c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556fd9a5075c std::function<>::operator()() at ??:0
@ 0x7f6ee393de06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ee393e6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6ee394003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556fd9a5075c std::function<>::operator()() at ??:0
I20240503 02:01:12.765309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.777606 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.783229 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.783463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.796362 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.797313 23337 ts_tablet_manager.cc:902] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:01:12.797660 14519 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:12.797742 23337 tablet_copy_client.cc:308] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:01:12.798296 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet a105660c12094c71b3a4b80e661d9308 from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:57713)
I20240503 02:01:12.798399 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet a105660c12094c71b3a4b80e661d9308 from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:57713: session id = 03ced8676740485e86ebe292f8ee4175-a105660c12094c71b3a4b80e661d9308
I20240503 02:01:12.799376 13055 tablet_copy_source_session.cc:205] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:01:12.800050 23337 ts_tablet_manager.cc:1875] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:01:12.802320 23337 ts_tablet_manager.cc:1888] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:01:12.802476 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:12.803704 23337 tablet_copy_client.cc:783] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 0 data blocks...
I20240503 02:01:12.803861 23337 tablet_copy_client.cc:647] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:01:12.804968 23337 tablet_copy_client.cc:523] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:01:12.806108 23337 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:01:12.810475 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.825752 23347 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:01:12.826195 23347 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:01:12.826395 23347 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.102s user 0.027s sys 0.006s
I20240503 02:01:12.827198 23347 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:01:12.830782 23347 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:01:12.830916 23347 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:12.831293 23347 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:01:12.831903 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-a105660c12094c71b3a4b80e661d9308 received from {username='slave'} at 127.0.99.134:37179
I20240503 02:01:12.831980 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-a105660c12094c71b3a4b80e661d9308 on tablet a105660c12094c71b3a4b80e661d9308 with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:12.832707 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f6ee392d807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ee416b6db start_thread at ??:0
@ 0x7f6ee16a371f clone at ??:0
I20240503 02:01:12.888535 14665 raft_consensus.cc:1203] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:01:12.891607 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.896852 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:12.908565 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.908931 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.909860 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.911940 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:12.928241 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:12.951303 23337 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:01:12.951720 23337 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:01:12.951936 23337 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.146s user 0.026s sys 0.003s
I20240503 02:01:12.952414 23337 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:01:12.952715 23337 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:01:12.952831 23337 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:12.954383 23337 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:01:12.955016 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-a105660c12094c71b3a4b80e661d9308 received from {username='slave'} at 127.0.99.132:57713
I20240503 02:01:12.955097 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-a105660c12094c71b3a4b80e661d9308 on tablet a105660c12094c71b3a4b80e661d9308 with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:01:12.987213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:12.994995 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.001986 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.006275 13167 raft_consensus.cc:1203] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.9->[3.10-3.10] Dedup: 3.10->[]
I20240503 02:01:13.011211 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.013566 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.035840 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.048324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.050597 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.051486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.052232 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.052601 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.064316 13038 raft_consensus.cc:1203] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240503 02:01:13.079937 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.100683 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.110985 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.199616 14665 raft_consensus.cc:1203] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:01:13.200994 23612 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.202365 14665 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.203646 14933 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.205014 12625 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.230878 14802 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:13.232616 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:01:13.232666 14665 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 02:01:13.232749 14736 consensus_peers.cc:485] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007feed148fb44e9a6fe9d449b6d8f9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.233124 23634 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:13.233268 22965 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:13.235105 22965 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:13.235499 12907 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:13.236340 12712 connection.cc:535] client connection to 127.0.99.133:37055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.236464 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.236918 14870 connection.cc:535] client connection to 127.0.99.133:37055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.237027 14870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.238608 12974 connection.cc:535] client connection to 127.0.99.133:37055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.238759 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.239429 13105 connection.cc:535] client connection to 127.0.99.133:37055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.239534 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:13.239729 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.239825 12632 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:13.240851 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.241530 14665 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:13.253072 23639 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:13.237526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.99.129:46415)
0503 02:01:13.237720 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:13.237736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:13.237818 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.252589 (+ 14771us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":48,"threads_started":1}
W20240503 02:01:13.253628 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:13.253720 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:13.253785 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:13.254001 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:13.254072 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.257356 13167 raft_consensus.cc:1203] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 02:01:13.257513 13167 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
W20240503 02:01:13.259390 14870 consensus_peers.cc:485] T b4b734308d3b42958e3a2830a83d44e2 P 558837c679f246dcb3241d66a37d70f1 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.292117 13167 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:13.294132 14665 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:13.294479 14933 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:01:13.294561 21261 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:01:13.294922 21261 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:01:13.299230 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.300871 21261 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:13.303997 14933 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:13.306546 12632 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:01:13.308686 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:13.309664 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.310838 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8292c06377f445849acdc22d08bd612d with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54816
I20240503 02:01:13.312650 23736 tablet_replica.cc:330] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:01:13.312865 23736 raft_consensus.cc:2229] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240503 02:01:13.312891 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.313042 23736 raft_consensus.cc:2258] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:13.313578 23736 ts_tablet_manager.cc:1875] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:13.316663 14665 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:01:13.326845 23736 ts_tablet_manager.cc:1888] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:01:13.327014 23736 log.cc:1196] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8292c06377f445849acdc22d08bd612d
I20240503 02:01:13.337872 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet 8292c06377f445849acdc22d08bd612d (table rebalance_test_table_3 [id=bf094dc6792f4586844a8fa66aa149a1]) successfully deleted
W20240503 02:01:13.391304 14870 consensus_peers.cc:485] T 86e314466c01488bb661c596c405db51 P 558837c679f246dcb3241d66a37d70f1 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:13.419095 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:13.443236 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.449321 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.452499 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.452963 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.455309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.463261 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.463550 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.485594 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.493085 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.494793 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.513127 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.519865 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.527804 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.527976 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.552184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.552385 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.558535 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.561779 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.562820 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.581095 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.601512 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.601819 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.608150 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.619287 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.625306 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.633198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.636802 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.658116 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.693434 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.700686 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.702744 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.720930 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:13.724086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:13.728178 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.746117 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.758189 14984 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:00:57.445137 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.445401 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.449476 14992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:00:58.514091 14992 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:00:59.027850 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.027976 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.028010 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.028054 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.028095 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.028180 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.028241 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.028302 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.028343 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.028420 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.028473 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.028545 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.028605 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.028647 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.028676 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.074651 14992 rebalancer_tool.cc:1084] tablet b4b734308d3b42958e3a2830a83d44e2: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move ignored: Illegal state: tablet b4b734308d3b42958e3a2830a83d44e2: replica 9e39ab9b64544988b968015fe9fa06e6 is already present
I20240503 02:00:59.074770 14992 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:59.317911 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.317991 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.318090 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.318179 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.318218 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.318256 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.318293 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.318331 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.318369 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.318403 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.318436 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.318475 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.318508 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.318573 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.318814 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.318892 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.318928 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.318957 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.318995 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.319063 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.430153 14992 rebalancer_tool.cc:1074] tablet 141097686fe24c8f9956dcc1ad1d253c: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:00:59.559727 14992 rebalancer_tool.cc:1074] tablet 34e88e69beb64036bb676b771dabd2a4: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:00:59.825457 14992 rebalancer_tool.cc:1074] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:00:59.917047 14992 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:00.112965 14992 rebalancer_tool.cc:1074] tablet f5767bdfcc6249578f7c3ada3c08da6e: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:00.324247 14992 rebalancer_tool.cc:1084] tablet 04582c3f86d744ccbc96eec50a10994f: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 02:01:00.324326 14992 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:00.940521 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.940647 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.940711 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.940974 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.941053 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.941093 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.941131 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.941169 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.941212 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.941246 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.941279 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.941346 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:00.972456 14992 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:01.151502 14992 rebalancer_tool.cc:1074] tablet da7d3c59eeb949ec84db013f24e656b0: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:01.214350 14992 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:01.225378 14992 rebalancer_tool.cc:1084] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Illegal state: tablet 097947dce87b409ca4a57cb752fdd5e2: replica 558837c679f246dcb3241d66a37d70f1 is already present
I20240503 02:01:01.225461 14992 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.667572 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.667694 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.667783 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.667918 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.667979 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.668033 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.668080 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.668187 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.668228 14992 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:01.731114 14992 rebalancer_tool.cc:1074] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:01.748210 14992 rebalancer_tool.cc:1074] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:01.787142 14992 rebalancer_tool.cc:1074] tablet 13a95f88ac1c459ba01be0743297f696: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:06.345881 14992 rebalancer_tool.cc:1123] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move is abandoned: Not found: Server with UUID 96b8f7d7c1ce4e2cad5af6caabfd8c4a not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:08.535259 14992 rebalancer_tool.cc:1136] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:08.543267 14992 rebalancer_tool.cc:1074] tablet 063a7f2ed28a455697b000f52a4343a3: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:08.550060 14992 rebalancer_tool.cc:1136] tablet 13a95f88ac1c459ba01be0743297f696: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:08.556869 14992 rebalancer_tool.cc:1136] tablet 007feed148fb44e9a6fe9d449b6d8f9d: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:08.579625 14992 rebalancer_tool.cc:1136] tablet da7d3c59eeb949ec84db013f24e656b0: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:08.607383 14992 rebalancer_tool.cc:1136] tablet 34e88e69beb64036bb676b771dabd2a4: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:08.623384 14992 rebalancer_tool.cc:1136] tablet 141097686fe24c8f9956dcc1ad1d253c: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:09.333071 14992 rebalancer_tool.cc:1136] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:09.364171 14992 rebalancer_tool.cc:1136] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:09.447952 14992 rebalancer_tool.cc:1136] tablet f5767bdfcc6249578f7c3ada3c08da6e: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:10.060581 14992 rebalancer_tool.cc:1074] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:10.076627 14992 rebalancer_tool.cc:1074] tablet 86e314466c01488bb661c596c405db51: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:10.090024 14992 rebalancer_tool.cc:1074] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:10.104724 14992 rebalancer_tool.cc:1074] tablet 490776bd408a48a9bcbbc7a501baa8ce: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:10.143200 14992 rebalancer_tool.cc:1136] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:10.866986 14992 rebalancer_tool.cc:1136] tablet 86e314466c01488bb661c596c405db51: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:10.889406 14992 rebalancer_tool.cc:1136] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:11.030158 14992 rebalancer_tool.cc:1136] tablet 063a7f2ed28a455697b000f52a4343a3: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:11.105201 14992 rebalancer_tool.cc:1136] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:11.120806 14992 rebalancer_tool.cc:1074] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:11.148274 14992 rebalancer_tool.cc:1074] tablet 1c464b6e30bb414bbe70ed030c5f481c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:11.159919 14992 rebalancer_tool.cc:1074] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:11.186041 14992 rebalancer_tool.cc:1136] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:11.206101 14992 rebalancer_tool.cc:1136] tablet 1c464b6e30bb414bbe70ed030c5f481c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:11.235054 14992 rebalancer_tool.cc:1136] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:11.245956 14992 rebalancer_tool.cc:1136] tablet 490776bd408a48a9bcbbc7a501baa8ce: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:11.963369 14992 rebalancer_tool.cc:1074] tablet 13a95f88ac1c459ba01be0743297f696: '754fc04ea9914073a56c0cf164f63240' -> '03ced8676740485e86ebe292f8ee4175' move scheduled
I20240503 02:01:12.068087 14992 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:12.146000 14992 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '558837c679f246dcb3241d66a37d70f1' -> '22b582a5f4774861a5619c51198d39c5' move scheduled
I20240503 02:01:13.203631 14992 rebalancer_tool.cc:1074] tablet a105660c12094c71b3a4b80e661d9308: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:01:13.242470 14992 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:13.259202 14992 rebalancer_tool.cc:1074] tablet 79c5f35eb8b546de8015c5235913efce: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:13.263326 14992 rebalancer_tool.cc:1084] tablet f5767bdfcc6249578f7c3ada3c08da6e: '754fc04ea9914073a56c0cf164f63240' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.263402 14992 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): unacceptable health status UNAVAILABLE
W20240503 02:01:13.777479 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.795653 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:13.797349 14671 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:01:13.797569 14933 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:01:13.797991 23941 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:13.798419 23941 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:01:13.800552 23612 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.800892 14933 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.802341 14671 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.803673 12633 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 558837c679f246dcb3241d66a37d70f1 reported cstate change: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:01:13.806602 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet a105660c12094c71b3a4b80e661d9308 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:13.806951 23975 tablet_replica.cc:330] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:13.807075 23975 raft_consensus.cc:2229] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:13.807163 23975 raft_consensus.cc:2258] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:13.807919 23975 ts_tablet_manager.cc:1875] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:13.811621 14986 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:00:57.344228 14988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.344738 14988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.348945 14988 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:00:58.378947 14988 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:00:58.976644 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.976769 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.976852 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.976943 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.977044 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.977118 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.977164 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.977211 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.977270 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.977321 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.977421 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.977492 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.977562 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.977609 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.049130 14988 rebalancer_tool.cc:1074] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:00:59.118328 14988 rebalancer_tool.cc:1074] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:00:59.229236 14988 rebalancer_tool.cc:1074] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:00:59.444772 14988 rebalancer_tool.cc:1074] tablet fdbd697a90534ff3b7a5e17780207d88: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:00:59.589644 14988 rebalancer_tool.cc:1074] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:00:59.739779 14988 rebalancer_tool.cc:1084] tablet 86e314466c01488bb661c596c405db51: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '9e39ab9b64544988b968015fe9fa06e6' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:00:59.739883 14988 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:00.187590 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.187721 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.187928 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.188016 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.188529 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.188637 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.188715 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.188769 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.188827 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.188966 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.189132 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.189199 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.189246 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.189306 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.189383 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.189442 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.189502 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.189553 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.189601 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.190093 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.190161 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.190218 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.190264 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.220062 14988 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:00.418699 14988 rebalancer_tool.cc:1074] tablet 04582c3f86d744ccbc96eec50a10994f: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:00.617558 14988 rebalancer_tool.cc:1074] tablet c361f82c69ef477d9c3cfd65e07d7f81: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:00.795914 14988 rebalancer_tool.cc:1074] tablet 0f3bdd27c750447a87691c0f57a70ee1: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:00.995394 14988 rebalancer_tool.cc:1074] tablet da7d3c59eeb949ec84db013f24e656b0: 'bbba999bf95446a68b6586608b33ee16' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:01:01.184638 14988 rebalancer_tool.cc:1074] tablet 3b829f779c8a459597c47dc3e42a4c46: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:01.244295 14988 rebalancer_tool.cc:1074] tablet 6ad545d33bbc40458c75ed7c90d92ba7: '03ced8676740485e86ebe292f8ee4175' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:01.288061 14988 rebalancer_tool.cc:1074] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:01.381857 14988 rebalancer_tool.cc:1074] tablet 8292c06377f445849acdc22d08bd612d: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:01.544854 14988 rebalancer_tool.cc:1084] tablet 8108933516894d4190ef9a0cf6063be8: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move ignored: Illegal state: tablet 8108933516894d4190ef9a0cf6063be8: replica 754fc04ea9914073a56c0cf164f63240 is already present
I20240503 02:01:01.544962 14988 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.986804 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.986960 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.987005 14988 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:02.039260 14988 rebalancer_tool.cc:1074] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:02.048980 14988 rebalancer_tool.cc:1074] tablet 86e314466c01488bb661c596c405db51: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:02.069406 14988 rebalancer_tool.cc:1074] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:02.346467 14988 rebalancer_tool.cc:1074] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:04.839552 14988 rebalancer_tool.cc:1136] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:05.606854 14988 rebalancer_tool.cc:1136] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:06.214874 14988 rebalancer_tool.cc:1123] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move is abandoned: Not found: Server with UUID 96b8f7d7c1ce4e2cad5af6caabfd8c4a not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:07.196635 14988 rebalancer_tool.cc:1136] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:07.227141 14988 rebalancer_tool.cc:1136] tablet 04582c3f86d744ccbc96eec50a10994f: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:07.286521 14988 rebalancer_tool.cc:1136] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:07.301858 14988 rebalancer_tool.cc:1136] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:07.835898 14988 rebalancer_tool.cc:1136] tablet 3b829f779c8a459597c47dc3e42a4c46: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:07.849704 14988 rebalancer_tool.cc:1136] tablet da7d3c59eeb949ec84db013f24e656b0: 'bbba999bf95446a68b6586608b33ee16' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:07.873390 14988 rebalancer_tool.cc:1136] tablet 6ad545d33bbc40458c75ed7c90d92ba7: '03ced8676740485e86ebe292f8ee4175' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:08.604259 14988 rebalancer_tool.cc:1074] tablet 141097686fe24c8f9956dcc1ad1d253c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:08.613543 14988 rebalancer_tool.cc:1074] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:08.635699 14988 rebalancer_tool.cc:1074] tablet b4b734308d3b42958e3a2830a83d44e2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:09.322801 14988 rebalancer_tool.cc:1136] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:09.358043 14988 rebalancer_tool.cc:1136] tablet 141097686fe24c8f9956dcc1ad1d253c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:09.414836 14988 rebalancer_tool.cc:1136] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:09.423537 14988 rebalancer_tool.cc:1136] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:09.441427 14988 rebalancer_tool.cc:1136] tablet fdbd697a90534ff3b7a5e17780207d88: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:09.469100 14988 rebalancer_tool.cc:1136] tablet 0f3bdd27c750447a87691c0f57a70ee1: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:09.485576 14988 rebalancer_tool.cc:1136] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:10.405134 14988 rebalancer_tool.cc:1136] tablet c361f82c69ef477d9c3cfd65e07d7f81: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:10.444108 14988 rebalancer_tool.cc:1136] tablet 86e314466c01488bb661c596c405db51: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:10.485622 14988 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:10.505370 14988 rebalancer_tool.cc:1074] tablet 1c464b6e30bb414bbe70ed030c5f481c: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:10.513906 14988 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:10.631475 14988 rebalancer_tool.cc:1074] tablet 8108933516894d4190ef9a0cf6063be8: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:12.158072 14988 rebalancer_tool.cc:1136] tablet 1c464b6e30bb414bbe70ed030c5f481c: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:12.642390 14988 rebalancer_tool.cc:1136] tablet b4b734308d3b42958e3a2830a83d44e2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
W20240503 02:01:13.247498 23594 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:13.003253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.0.1:57160)
0503 02:01:13.025708 (+ 22455us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:13.025724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:13.025840 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.246852 (+221012us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":22384,"thread_start_us":62,"threads_started":1}
I20240503 02:01:13.353870 14988 rebalancer_tool.cc:1074] tablet 13a95f88ac1c459ba01be0743297f696: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:13.363473 14988 rebalancer_tool.cc:1084] tablet f5767bdfcc6249578f7c3ada3c08da6e: 'bbba999bf95446a68b6586608b33ee16' -> '' move ignored: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.363576 14988 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): unacceptable health status UNAVAILABLE
I20240503 02:01:13.815845 23975 ts_tablet_manager.cc:1888] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:01:13.815995 23975 log.cc:1196] T a105660c12094c71b3a4b80e661d9308 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a105660c12094c71b3a4b80e661d9308
I20240503 02:01:13.816710 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet a105660c12094c71b3a4b80e661d9308 (table rebalance_test_table_9 [id=319c91db9ef042e9a8e1ad6fb0b8cd24]) successfully deleted
I20240503 02:01:13.831923 24002 ts_tablet_manager.cc:902] T 007feed148fb44e9a6fe9d449b6d8f9d P 558837c679f246dcb3241d66a37d70f1: Initiating tablet copy from peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
F20240503 02:01:13.832183 24002 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-007feed148fb44e9a6fe9d449b6d8f9d.queue_time_us, tablet-download-pool-007feed148fb44e9a6fe9d449b6d8f9d.queue_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.queue_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.queue_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_cpu_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_cpu_time_us) (tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_wall_time_us, tablet-download-pool-04582c3f86d744ccbc96eec50a10994f.run_wall_time_us) (tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.queue_time_us, tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.queue_time_us) (tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.run_cpu_time_us, tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.run_cpu_time_us) (tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.run_wall_time_us, tablet-download-pool-097947dce87b409ca4a57cb752fdd5e2.run_wall_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.queue_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.queue_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_cpu_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_cpu_time_us) (tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_wall_time_us, tablet-download-pool-0f3bdd27c750447a87691c0f57a70ee1.run_wall_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.queue_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.queue_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_cpu_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_cpu_time_us) (tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_wall_time_us, tablet-download-pool-1c464b6e30bb414bbe70ed030c5f481c.run_wall_time_us) (tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.queue_time_us, tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.queue_time_us) (tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.run_cpu_time_us, tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.run_cpu_time_us) (tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.run_wall_time_us, tablet-download-pool-3b829f779c8a459597c47dc3e42a4c46.run_wall_time_us) (tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.queue_time_us, tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.queue_time_us) (tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.run_cpu_time_us, tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.run_cpu_time_us) (tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.run_wall_time_us, tablet-download-pool-490776bd408a48a9bcbbc7a501baa8ce.run_wall_time_us) (tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.queue_time_us, tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.queue_time_us) (tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.run_cpu_time_us, tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.run_cpu_time_us) (tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.run_wall_time_us, tablet-download-pool-613074d3c4514bd5b64a06c59af09a56.run_wall_time_us) (tablet-download-pool-8292c06377f445849acdc22d08bd612d.queue_time_us, tablet-download-pool-8292c06377f445849acdc22d08bd612d.queue_time_us) (tablet-download-pool-8292c06377f445849acdc22d08bd612d.run_cpu_time_us, tablet-download-pool-8292c06377f445849acdc22d08bd612d.run_cpu_time_us) (tablet-download-pool-8292c06377f445849acdc22d08bd612d.run_wall_time_us, tablet-download-pool-8292c06377f445849acdc22d08bd612d.run_wall_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.queue_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.queue_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.run_cpu_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.run_cpu_time_us) (tablet-download-pool-86e314466c01488bb661c596c405db51.run_wall_time_us, tablet-download-pool-86e314466c01488bb661c596c405db51.run_wall_time_us) (tablet-download-pool-a105660c12094c71b3a4b80e661d9308.queue_time_us, tablet-download-pool-a105660c12094c71b3a4b80e661d9308.queue_time_us) (tablet-download-pool-a105660c12094c71b3a4b80e661d9308.run_cpu_time_us, tablet-download-pool-a105660c12094c71b3a4b80e661d9308.run_cpu_time_us) (tablet-download-pool-a105660c12094c71b3a4b80e661d9308.run_wall_time_us, tablet-download-pool-a105660c12094c71b3a4b80e661d9308.run_wall_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.queue_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.queue_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_cpu_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_cpu_time_us) (tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_wall_time_us, tablet-download-pool-ae1f9c8efd2040d6842c6b9f7c6173aa.run_wall_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.queue_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.queue_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_cpu_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_cpu_time_us) (tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_wall_time_us, tablet-download-pool-b4b734308d3b42958e3a2830a83d44e2.run_wall_time_us) (tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.queue_time_us, tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.queue_time_us) (tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.run_cpu_time_us, tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.run_cpu_time_us) (tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.run_wall_time_us, tablet-download-pool-d86b733f38fd4f08bd40865b99e2e7b2.run_wall_time_us) (tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.queue_time_us, tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.queue_time_us) (tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.run_cpu_time_us, tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.run_cpu_time_us) (tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.run_wall_time_us, tablet-download-pool-da7d3c59eeb949ec84db013f24e656b0.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f2abe99bdcd google::LogMessage::Fail() at ??:0
@ 0x7f2abe99fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2abe99b7cc google::LogMessage::Flush() at ??:0
@ 0x7f2abe99cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2abfe05044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f2abfde9f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f2abfde7efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f2ac299a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:01:13.868311 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f2ac29a3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f2ac2a2476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:01:13.876818 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7f2ac2a384f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2ac2a23226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f2ac2a32c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55dde14cf75c std::function<>::operator()() at ??:0
I20240503 02:01:13.886588 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f2abfdece06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 02:01:13.888177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.895912 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f2abfded6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 02:01:13.898974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f2abfdef03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55dde14cf75c std::function<>::operator()() at ??:0
@ 0x7f2abfddc807 kudu::Thread::SuperviseThread() at ??:0
I20240503 02:01:13.904503 14914 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f2ac061a6db start_thread at ??:0
@ 0x7f2abdb5271f clone at ??:0
I20240503 02:01:13.935981 23612 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:13.936087 23612 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:13.936478 23612 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:01:13.937973 14802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "754fc04ea9914073a56c0cf164f63240" is_pre_election: true
W20240503 02:01:13.938488 12974 leader_election.cc:334] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.938548 12974 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5; no voters: 754fc04ea9914073a56c0cf164f63240, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:13.938772 23612 raft_consensus.cc:2726] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:13.989720 14782 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:01:13.991039 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:13.995913 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.007694 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.017092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.028522 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.031538 23640 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.031656 23640 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:14.032157 23640 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:01:14.033661 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
I20240503 02:01:14.033798 13038 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 2.
I20240503 02:01:14.034211 14738 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 754fc04ea9914073a56c0cf164f63240; no voters:
I20240503 02:01:14.040935 23640 raft_consensus.cc:2781] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:01:14.041003 23640 raft_consensus.cc:482] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.041039 23640 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:14.042093 23640 raft_consensus.cc:504] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:14.042464 23640 leader_election.cc:288] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 election: Requested vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:14.042572 14738 leader_election.cc:334] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.043181 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5767bdfcc6249578f7c3ada3c08da6e" candidate_uuid: "754fc04ea9914073a56c0cf164f63240" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5"
I20240503 02:01:14.043301 13038 raft_consensus.cc:3037] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:14.044340 13038 raft_consensus.cc:2454] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 754fc04ea9914073a56c0cf164f63240 in term 3.
I20240503 02:01:14.044674 14738 leader_election.cc:302] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 754fc04ea9914073a56c0cf164f63240; no voters:
W20240503 02:01:14.045892 14738 leader_election.cc:334] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.046110 23640 raft_consensus.cc:2781] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:14.046255 23640 raft_consensus.cc:686] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 3 LEADER]: Becoming Leader. State: Replica: 754fc04ea9914073a56c0cf164f63240, State: Running, Role: LEADER
I20240503 02:01:14.046408 23640 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:14.048525 12632 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 reported cstate change: term changed from 2 to 3, leader changed from 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) to 754fc04ea9914073a56c0cf164f63240 (127.0.99.135). New cstate: current_term: 3 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:14.058629 23132 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.058758 23132 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:14.059259 23132 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:14.060113 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:14.060511 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.060601 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:14.060750 23132 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:14.098709 13016 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.105566 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.110390 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.113823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.115345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.117406 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.165721 23729 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.165886 23729 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:14.166455 23729 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:01:14.166838 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "22b582a5f4774861a5619c51198d39c5" is_pre_election: true
W20240503 02:01:14.167831 12843 leader_election.cc:334] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.167909 12843 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters: 22b582a5f4774861a5619c51198d39c5, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:14.168052 23729 raft_consensus.cc:2726] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:14.211015 23612 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.211187 23612 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:14.211786 23612 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:01:14.212205 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" is_pre_election: true
I20240503 02:01:14.212369 12905 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 2.
I20240503 02:01:14.212692 12973 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
W20240503 02:01:14.212878 12974 leader_election.cc:334] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.212915 23612 raft_consensus.cc:2781] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:01:14.212981 23612 raft_consensus.cc:482] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:14.213042 23612 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:01:14.214474 23612 raft_consensus.cc:504] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
W20240503 02:01:14.215219 12710 connection.cc:535] client connection to 127.0.99.136:41447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:14.215303 13103 connection.cc:535] client connection to 127.0.99.136:41447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:14.215363 12841 connection.cc:535] client connection to 127.0.99.136:41447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:01:14.215808 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e88e69beb64036bb676b771dabd2a4" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a"
I20240503 02:01:14.215924 12905 raft_consensus.cc:3037] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:01:14.216464 12972 connection.cc:535] client connection to 127.0.99.136:41447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:01:14.217039 12905 raft_consensus.cc:2454] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 3.
W20240503 02:01:14.217474 23639 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:14.215831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.99.129:39951)
0503 02:01:14.216048 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:14.216062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:14.216146 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.217381 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":136}
W20240503 02:01:14.217602 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.217659 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.217978 12973 leader_election.cc:302] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, 96b8f7d7c1ce4e2cad5af6caabfd8c4a; no voters:
W20240503 02:01:14.218168 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.218186 23974 raft_consensus.cc:2781] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:01:14.218259 23974 raft_consensus.cc:686] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
W20240503 02:01:14.218413 23650 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:14.216047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.99.132:60387)
0503 02:01:14.216263 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:14.216275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:14.216357 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.217883 (+ 1526us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122}
W20240503 02:01:14.218432 23728 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:14.216330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.99.135:40825)
0503 02:01:14.216508 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:14.216520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:14.216603 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.217920 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98}
I20240503 02:01:14.218397 23974 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
W20240503 02:01:14.218559 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:14.218705 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.218772 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.219089 23612 leader_election.cc:288] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 election: Requested vote from peers 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:14.220045 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.220459 12626 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 2 to 3, leader changed from 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } }
W20240503 02:01:14.222800 12974 leader_election.cc:334] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:14.245196 14665 consensus_queue.cc:237] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:01:14.246613 14802 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:01:14.246667 12905 raft_consensus.cc:1261] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer d23425fa2b5848c8b1a2b737d75e69eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:01:14.247020 22983 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:01:14.247223 24180 consensus_queue.cc:1035] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:01:14.249171 24183 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:14.249502 14802 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 754fc04ea9914073a56c0cf164f63240 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:14.252341 12626 catalog_manager.cc:5484] T 13a95f88ac1c459ba01be0743297f696 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 10 to 11, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New cstate: current_term: 3 leader_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:14.253211 12905 raft_consensus.cc:2932] T 13a95f88ac1c459ba01be0743297f696 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:01:14.254974 13149 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a95f88ac1c459ba01be0743297f696 with delete_type TABLET_DATA_TOMBSTONED (TS 03ced8676740485e86ebe292f8ee4175 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50056
I20240503 02:01:14.255254 23975 tablet_replica.cc:330] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: stopping tablet replica
I20240503 02:01:14.255407 23975 raft_consensus.cc:2229] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:01:14.255697 23975 raft_consensus.cc:2258] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:01:14.256333 23975 ts_tablet_manager.cc:1875] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:01:14.258236 23975 ts_tablet_manager.cc:1888] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240503 02:01:14.258293 23975 log.cc:1196] T 13a95f88ac1c459ba01be0743297f696 P 03ced8676740485e86ebe292f8ee4175: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/13a95f88ac1c459ba01be0743297f696
I20240503 02:01:14.258889 12620 catalog_manager.cc:4830] TS 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): tablet 13a95f88ac1c459ba01be0743297f696 (table rebalance_test_table_6 [id=60828393e9d5433c908a3ba9783dd7a0]) successfully deleted
W20240503 02:01:14.260576 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.281886 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:14.288312 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.291193 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:14.302249 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.306143 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.306793 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.316398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.321818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.323715 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:14.334398 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.335956 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:14.399727 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.401283 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.402344 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.403746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.410725 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.417557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.419497 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.436358 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:01:14.436993 23640 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:01:14.439220 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.441529 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:14.464766 14985 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:00:57.375432 14989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.375722 14989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.381000 14989 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:00:58.426338 14989 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:00:58.946833 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.946971 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.947079 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.947139 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.947289 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.947335 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.947415 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.947469 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.947520 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.947556 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.947604 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.947643 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.947682 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.947731 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:00:59.036324 14989 rebalancer_tool.cc:1074] tablet 063a7f2ed28a455697b000f52a4343a3: 'bbba999bf95446a68b6586608b33ee16' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:00:59.111486 14989 rebalancer_tool.cc:1074] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:00:59.167488 14989 rebalancer_tool.cc:1074] tablet 1c464b6e30bb414bbe70ed030c5f481c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:00:59.316231 14989 rebalancer_tool.cc:1084] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> '9e39ab9b64544988b968015fe9fa06e6' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 02:00:59.316334 14989 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:59.790153 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.790267 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.790321 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.790359 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.790421 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.790473 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.790517 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.790560 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.790601 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.790652 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.790704 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.790741 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.790807 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.790845 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.790881 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.790918 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.790956 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.791002 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.791041 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.973380 14989 rebalancer_tool.cc:1074] tablet f5767bdfcc6249578f7c3ada3c08da6e: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:00.061617 14989 rebalancer_tool.cc:1074] tablet 13a95f88ac1c459ba01be0743297f696: '03ced8676740485e86ebe292f8ee4175' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:01:00.142907 14989 rebalancer_tool.cc:1084] tablet 86e314466c01488bb661c596c405db51: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Illegal state: tablet 86e314466c01488bb661c596c405db51: replica 558837c679f246dcb3241d66a37d70f1 is already present
I20240503 02:01:00.142987 14989 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:00.726393 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.726537 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.726580 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.726608 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.726672 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.726722 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.726756 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.726788 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.726820 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.726855 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.726881 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.726930 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.726981 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.756080 14989 rebalancer_tool.cc:1074] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:00.983950 14989 rebalancer_tool.cc:1074] tablet 3b829f779c8a459597c47dc3e42a4c46: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:01.032162 14989 rebalancer_tool.cc:1074] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:01.107889 14989 rebalancer_tool.cc:1074] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:01.238242 14989 rebalancer_tool.cc:1074] tablet 0f3bdd27c750447a87691c0f57a70ee1: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:01.256134 14989 rebalancer_tool.cc:1074] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:01.266865 14989 rebalancer_tool.cc:1084] tablet da7d3c59eeb949ec84db013f24e656b0: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move ignored: Illegal state: tablet da7d3c59eeb949ec84db013f24e656b0: replica d23425fa2b5848c8b1a2b737d75e69eb is already present
I20240503 02:01:01.266952 14989 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.639394 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.647850 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.647933 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.647975 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.648030 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.648066 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.648102 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.648146 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.648183 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.648218 14989 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:01.812391 14989 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:01.845446 14989 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:06.794014 14989 rebalancer_tool.cc:1136] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:07.392434 14989 rebalancer_tool.cc:1136] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:09.002275 14989 rebalancer_tool.cc:1074] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:09.119454 14989 rebalancer_tool.cc:1136] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:09.138350 14989 rebalancer_tool.cc:1136] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:09.150606 14989 rebalancer_tool.cc:1136] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:09.165921 14989 rebalancer_tool.cc:1136] tablet 0f3bdd27c750447a87691c0f57a70ee1: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:09.204957 14989 rebalancer_tool.cc:1136] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:09.245204 14989 rebalancer_tool.cc:1136] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:09.286221 14989 rebalancer_tool.cc:1136] tablet 3b829f779c8a459597c47dc3e42a4c46: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:09.327075 14989 rebalancer_tool.cc:1136] tablet 063a7f2ed28a455697b000f52a4343a3: 'bbba999bf95446a68b6586608b33ee16' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:09.336390 14989 rebalancer_tool.cc:1136] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:09.361867 14989 rebalancer_tool.cc:1136] tablet 1c464b6e30bb414bbe70ed030c5f481c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:10.488862 14989 rebalancer_tool.cc:1074] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:10.510272 14989 rebalancer_tool.cc:1074] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:10.531159 14989 rebalancer_tool.cc:1074] tablet a105660c12094c71b3a4b80e661d9308: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:10.546191 14989 rebalancer_tool.cc:1074] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:10.555562 14989 rebalancer_tool.cc:1074] tablet b4b734308d3b42958e3a2830a83d44e2: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:11.427039 14989 rebalancer_tool.cc:1136] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:12.125270 14989 rebalancer_tool.cc:1136] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:12.143246 14989 rebalancer_tool.cc:1136] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:12.159929 14989 rebalancer_tool.cc:1136] tablet f5767bdfcc6249578f7c3ada3c08da6e: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:12.637667 14989 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:12.658895 14989 rebalancer_tool.cc:1136] tablet 007feed148fb44e9a6fe9d449b6d8f9d: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
W20240503 02:01:13.245021 23444 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:12.988778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.0.1:57156)
0503 02:01:12.998261 (+ 9483us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:12.998275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:12.998330 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.244239 (+245909us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":9413}
W20240503 02:01:14.216606 24098 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:13.977998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.0.1:48446)
0503 02:01:13.979707 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:13.979718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:13.979817 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.216441 (+236624us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1650,"thread_start_us":98,"threads_started":1}
I20240503 02:01:14.262117 14989 rebalancer_tool.cc:1123] tablet b4b734308d3b42958e3a2830a83d44e2: '03ced8676740485e86ebe292f8ee4175' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:14.268311 14989 rebalancer_tool.cc:1136] tablet a105660c12094c71b3a4b80e661d9308: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:14.274446 14989 rebalancer_tool.cc:1136] tablet 13a95f88ac1c459ba01be0743297f696: '03ced8676740485e86ebe292f8ee4175' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
Illegal state: tablet server 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): unacceptable health status UNAVAILABLE
I20240503 02:01:14.505369 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.506146 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.508082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.509225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.509711 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.520748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.609959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.613457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.620329 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.623629 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.624368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.624862 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.685118 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.686676 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.686848 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.699014 12905 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:01:14.699522 23612 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:01:14.701737 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.703534 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.714413 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:14.730243 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:14.772269 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:14.810869 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:14.926496 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.927978 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.934911 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:14.935300 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:14.935945 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:14.954406 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:14.992908 23640 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:14.993050 23640 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:14.993525 23640 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:14.995241 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:14.995376 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:14.995429 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:14.995641 23640 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:15.025563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.027129 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.029708 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:15.032685 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.051743 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.055989 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.098215 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.098953 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.099807 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.099807 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.104426 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.120532 24396 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:15.120679 24396 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:15.121121 24396 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:15.122157 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:15.122879 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:15.122948 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:15.123260 24396 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:15.124300 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:15.343699 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:15.345856 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.346159 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.361518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.366014 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:15.374707 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:15.376230 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.379099 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:15.389168 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:15.455433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.461643 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.461737 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.462884 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.464601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.467911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:15.491165 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:15.516693 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:15.561784 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.562659 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.567394 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:15.573763 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.584743 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.586175 24388 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:15.586303 24388 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:15.586769 24388 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:15.587277 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:15.587821 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:15.587890 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:15.588102 24388 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:15.597612 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:15.626598 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:15.648087 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:15.673885 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.679394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:15.681022 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.682518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.682929 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:15.692198 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:15.692638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:15.796031 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:15.847927 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:15.885241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:15.886693 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.887847 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.891103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:15.898366 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:15.916249 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:15.957644 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:16.011715 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.020999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:16.024027 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.030579 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.037068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.038410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.052592 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:16.054073 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.065523 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.072305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.073371 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.099711 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:16.120141 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.181430 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.330853 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.360244 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:16.369668 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:16.380335 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.382411 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.384526 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:16.409978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.414387 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.418015 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.418843 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.433395 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.441140 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.445163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.447649 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:16.462083 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.468072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:16.498574 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.499927 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.547323 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.549690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.552719 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.556421 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.556509 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.577476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:16.583679 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.587564 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.628367 23640 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:16.628475 23640 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:16.628870 23640 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:16.630017 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:16.630113 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:16.630193 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:16.630316 23640 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:16.750459 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.777554 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.781080 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.787042 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.796936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.798513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:16.801630 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:16.825109 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:16.868100 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:16.881045 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.884033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.886587 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.887943 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:16.893841 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.906266 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:16.922740 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:16.933739 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.984998 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:16.988106 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:16.988550 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.989264 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:16.989286 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:16.993180 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.002200 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:17.009035 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:17.009186 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:17.038333 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:01:17.101403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.102187 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.105907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.106685 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.129283 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.133170 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:17.178831 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:01:17.218492 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:17.285260 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:17.293705 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.294411 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.296976 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.297482 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.307988 25023 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:17.308094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.308115 25023 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:17.308534 25023 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:17.313537 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:17.313666 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:17.313727 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:17.314406 25023 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:17.320173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.409677 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.409893 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.411065 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.412278 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.422207 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.423702 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.480717 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.481097 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.483697 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.485291 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.491899 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.492398 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.611624 24388 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:17.611768 24388 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:17.612147 24388 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:17.612919 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:17.613094 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:17.613145 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:17.613268 24388 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:17.630847 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:17.704525 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:17.724124 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.725498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.728909 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.731057 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.744287 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.747042 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.841296 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.848167 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:17.849064 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:17.857445 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.858592 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:17.875537 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:17.880334 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.900301 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.900671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.912338 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:17.926935 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:17.929230 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:17.930291 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:17.936667 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:17.957455 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:17.982827 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:18.109298 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:18.152036 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:18.191354 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.194507 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.199236 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.199491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:18.199782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.211586 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:18.236361 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:18.247012 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240503 02:01:18.264636 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:01:18.266028 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:18.270715 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:01:18.290293 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.290982 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.291345 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.298162 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:18.300583 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.304901 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:18.398964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:18.399884 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:01:18.401684 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.405529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:18.406623 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:18.413702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.422791 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:18.455354 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:01:18.456173 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:18.476758 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:18.508951 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.512383 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.513029 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:18.513250 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:18.513803 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.520860 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.533569 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:18.677804 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:18.704104 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:18.707851 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.710405 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.715114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.715823 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:18.716885 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:18.718636 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:18.776894 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:18.834936 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.837347 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.840459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.846479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:18.853511 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.874020 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:18.896878 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:18.899348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:18.901288 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.903659 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:18.909497 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:18.914350 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:18.922573 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:18.927436 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:19.049978 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.076342 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.166662 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:19.172621 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.173573 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.184387 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:19.192267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.201640 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.212599 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:19.230020 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.258224 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:19.262017 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.262636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.271486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.275219 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.275576 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.276196 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:19.340767 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:19.368430 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.368623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:19.368835 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.371843 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.376888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.390344 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:19.398874 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.420269 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.421701 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.452666 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:19.493989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.496191 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.500819 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.501964 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.503141 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:19.504563 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:19.508141 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:19.567984 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:01:19.677489 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:19.682595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:19.684880 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.687179 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.688783 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.689352 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:19.691884 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:19.706981 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.797750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.801366 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.802244 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.803362 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.822760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.823138 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:19.842468 23640 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:19.842582 23640 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:19.842948 23640 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:19.843953 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:19.844106 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:19.844178 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:19.844298 23640 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:19.871667 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.876534 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:19.877736 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:19.884826 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:19.891386 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:19.898305 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:01:19.902976 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:20.033109 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:20.124495 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.124770 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:20.125051 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.127919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:20.129835 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.131265 25227 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:20.131366 25227 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:20.131784 25227 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:01:20.141413 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:20.145108 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:20.145349 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:20.145431 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:20.145735 25227 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:20.240669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.242139 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.251153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:20.252830 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.254129 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.261082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.268522 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.275516 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.280529 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:20.286332 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:20.288625 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.312132 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:20.322402 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:20.451023 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:20.572675 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:20.607460 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.609119 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:20.618633 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:20.619565 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:20.620296 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.620438 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.637648 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:20.645885 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:20.661299 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:20.676458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.676527 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.677026 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.682323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:20.682317 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.691368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:20.692368 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:20.782918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.785171 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.785214 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:20.785701 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.788081 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:20.788327 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:20.851233 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:20.887714 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:20.930891 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.931900 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.935914 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:20.939913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:20.942155 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:20.944442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:20.967235 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.008199 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:21.086979 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.095047 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.095252 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.096671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.100281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.113075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:21.146303 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:21.188457 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:21.195919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.196728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.204331 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:21.207952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.217662 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.229022 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.279515 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:21.280766 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:21.280992 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.284910 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.290841 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.296324 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.309616 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:21.311666 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.349094 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:21.472014 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:21.519309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.519423 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.525616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.528043 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:21.535776 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.539304 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:21.564750 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.586705 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:21.622524 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.627012 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.630167 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.630285 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.638232 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.650164 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:21.678721 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.687916 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.692786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.695910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.699059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.713651 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:21.736048 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.842929 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.843324 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.849556 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.854863 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:21.971570 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:21.972095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.980006 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:21.980367 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:21.984274 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:21.991860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:21.998402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.047927 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.050287 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.059530 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:22.060164 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:22.060495 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:22.061254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.062656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:22.097448 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:22.125546 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:22.151333 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.153512 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:22.153606 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.157212 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:22.164106 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.165524 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:22.169726 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:22.462335 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.464270 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.466670 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:22.467829 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.470836 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:22.471474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:22.478283 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:01:22.537374 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:22.542166 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:22.550290 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:01:22.565611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.570852 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.572165 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.573174 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:22.573789 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:22.588063 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:22.645483 26581 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:22.645606 26581 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:22.646042 26581 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:22.646736 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:22.647048 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:22.647125 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:22.647270 26581 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:22.880430 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.883359 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.885340 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:22.885428 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:22.887035 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.887979 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:22.982975 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:22.985914 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:22.992753 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:22.993929 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:22.994747 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.001435 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:23.002085 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.013036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.075995 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:23.079066 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.204995 25227 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:23.205111 25227 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:23.205524 25227 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:23.206627 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:23.206862 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:23.206969 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:23.207140 25227 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:23.232705 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.260969 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.262611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.264228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:23.271415 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240503 02:01:23.277253 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.284184 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:23.284931 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:01:23.285897 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.301163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.306252 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:23.319155 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.353077 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.358546 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.359301 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.365381 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:23.365564 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.371357 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.411217 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:01:23.444684 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.475005 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.475930 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.476099 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.478402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.485037 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:23.500664 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:23.501951 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:23.505573 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:01:23.506342 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:23.517590 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:23.530109 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:23.568938 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:01:23.587225 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.591388 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.597592 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.599179 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.600725 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.609869 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:23.725291 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:23.727213 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.786592 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.787827 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.789541 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.789719 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.791433 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.800860 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:23.897073 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.900034 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.900981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.920140 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.925956 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.935205 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:23.936450 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.939126 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:23.945657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:23.951311 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:23.966022 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:23.977720 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:23.987454 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:23.992343 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:24.083822 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:24.128130 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:24.255015 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:24.267483 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:24.276322 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.280628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.281458 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.281836 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.286356 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.313215 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:24.320609 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:24.338822 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.343560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.348028 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.350272 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.353870 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:24.361737 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.449858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.450565 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.451117 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.452610 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:24.460669 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:24.463908 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.470063 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:24.505302 23640 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:24.505455 23640 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:24.505908 23640 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:24.507030 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:24.507138 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:24.507221 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:24.507345 23640 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:24.529843 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:24.541503 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:24.549360 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:24.583683 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:24.584237 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:24.614388 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:24.659797 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.660610 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.664098 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.671401 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.687857 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:24.688203 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.775964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.778676 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.784058 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.787856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.788117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.795099 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:24.885237 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.888976 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.891369 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:24.891980 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:24.895421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:24.900238 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.986129 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.987982 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.992396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:24.993695 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.002580 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:25.015051 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:25.017071 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:25.040074 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:01:25.198366 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:25.205351 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.205468 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.206980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.209467 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.209857 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.226902 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:25.308938 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:01:25.319721 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.324373 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.326541 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.330569 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.332038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.341558 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:25.361310 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:25.381345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.384586 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.385603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.391299 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.392851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.411612 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:25.434010 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:25.585820 26581 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:25.585997 26581 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:25.586431 26581 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:25.587126 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:25.587389 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:25.587474 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:25.587632 26581 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:25.617853 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:25.623481 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:25.664175 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:25.674247 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:25.685364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.685511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.686410 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.686998 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.688251 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.701009 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:25.707940 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:25.738030 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.743052 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.749338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.753638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.761512 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.765908 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:25.864807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.865291 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:25.872895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:25.877693 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:25.883936 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:25.885272 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:25.975004 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.028327 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:26.044776 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.045828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.046695 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.047219 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.047689 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.068745 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:26.103729 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.158329 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:26.173395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:26.175083 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:26.179543 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.184116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.191733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:26.192027 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:26.195000 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.195361 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:26.204344 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:26.281811 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.282528 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.285735 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.292258 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.294099 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.295975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.383617 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.387302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:26.387706 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:26.388020 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.389470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.396770 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.397924 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:26.438277 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.472677 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.580610 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:26.595957 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:26.598733 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.602555 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.602732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.602882 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.603958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.623605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.725976 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.728070 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.728523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:26.736527 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.741199 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.742060 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.749754 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.750828 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:26.770080 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:26.780274 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:26.795614 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:26.796517 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:26.797309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:26.822846 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.926815 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.938448 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:26.956635 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:27.008261 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:27.016275 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:27.040719 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:27.055452 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:01:27.061643 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:27.107677 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.111718 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.112025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.126080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.127193 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:27.129011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:27.132730 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:27.134557 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.139814 26581 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:27.139936 26581 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:27.140210 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:27.140301 26581 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:27.140810 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:01:27.140938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:27.147835 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:27.147922 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:27.148249 26581 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:27.148275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.150198 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.258230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.260164 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.264058 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:27.265192 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.287945 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:27.288249 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:27.389113 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:27.466290 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.471801 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:27.476389 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:01:27.479082 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:27.479940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.480440 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.489661 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.573958 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.575831 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.577980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.578349 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:27.585079 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:27.587502 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.591495 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:27.634459 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:27.681056 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.682875 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:27.684329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.684578 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:27.685084 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.691953 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.789778 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.790095 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:27.795925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.798955 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.806537 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:27.807636 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:27.859040 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:27.869397 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:27.997396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:27.998162 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.002399 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.002873 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:28.003608 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.015489 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:28.072564 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:28.095115 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.120795 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.122709 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.123054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.131085 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.133831 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:28.134369 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.137815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:28.159251 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:28.160118 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:28.166368 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.167021 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.176313 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.177260 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.195356 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:28.222503 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:28.348708 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:28.424331 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:01:28.480322 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.482131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:28.483953 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:01:28.485641 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.496999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.500797 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:28.507794 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:01:28.508702 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:28.509552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.533056 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.542192 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.542925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.552968 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:28.559939 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:28.567359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:28.568337 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:28.587211 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:28.624048 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.627635 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.629904 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.634135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.634728 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.639011 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.663353 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:28.674513 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240503 02:01:28.800972 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.806674 28032 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:28.806804 28032 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:28.807159 28032 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:28.808125 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:28.808458 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:28.808537 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:28.808665 28032 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:28.811364 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:28.812197 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:28.812214 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.865428 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.866408 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.867116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:28.869704 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.869720 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.882984 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:28.928090 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:28.966230 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.966418 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:28.970712 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:28.980334 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:28.981724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:28.990067 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:28.993793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.073966 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.076457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.076666 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.078723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.078753 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:29.083334 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:29.093986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.176282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.176522 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.181583 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.190840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.198267 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.199959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:29.225944 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:29.226552 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:29.387465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.388926 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.391420 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.396024 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.403914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.416966 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:29.474606 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:29.499547 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:29.503130 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.504163 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:29.509408 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:29.510823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.515502 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.518358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.518944 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:29.530035 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.536741 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.544690 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:29.555948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:29.559077 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:29.559890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.570696 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:29.581499 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:29.602272 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:29.683725 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:29.726917 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:29.738672 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:29.774158 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:01:29.906387 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.910055 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.910604 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.921707 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.923588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.938820 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.948511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:29.950176 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:29.955917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:29.974455 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:29.982630 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:29.983868 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:29.995079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:30.031042 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:30.038110 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:01:30.098002 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.098222 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.099895 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.107918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:30.108352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.111999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:30.112603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.308141 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.309075 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.309157 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.309736 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:30.309969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:30.319932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:30.363550 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.413259 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.414255 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.415479 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.415957 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:30.425184 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.429121 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:30.444135 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.516927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.521817 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.522040 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.522910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.526595 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:30.534643 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:30.557847 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:30.566395 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:30.623935 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.624644 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.627388 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.634145 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:30.640717 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:30.644052 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.647933 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:30.670001 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.832293 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.835739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.837756 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.840567 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:30.843533 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:30.853665 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:30.854302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.950186 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.958710 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.961726 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.964395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:30.964712 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:30.966392 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:30.985635 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:30.998081 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.000205 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:31.011348 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:31.012694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.014500 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:31.077632 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:31.159512 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:31.189878 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:31.249250 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.320070 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.330210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.333225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.334935 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.346334 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.346627 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:31.355036 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.358107 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.358280 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:31.361359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.374020 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:31.378473 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:31.392068 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.417392 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:31.439280 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.484530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.486496 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.488387 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:31.489660 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.491098 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.500653 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:31.556238 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:31.594102 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:31.703613 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.706581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.708041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.710783 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:31.711074 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.720503 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:31.729764 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:31.752480 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:31.812224 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.813933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.817884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.822685 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:31.824103 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.839445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.841984 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:31.848739 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:31.923205 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:31.924335 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:31.930135 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:31.934420 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:31.936615 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:31.936935 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:31.948668 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:31.949148 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:31.970135 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:31.970342 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:31.981949 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:32.030216 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.030815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.034253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.047299 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:32.048580 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.058722 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.064605 28679 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:32.064733 28679 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:32.065138 28679 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:32.066574 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:32.067183 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:32.067253 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:32.067376 28679 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:32.242540 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.242998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.243796 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.244653 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:32.264873 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.265283 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:32.311143 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:32.330444 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:32.358808 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:01:32.361523 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.368283 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.368590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.369612 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:32.381314 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:32.382727 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.385324 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:32.405474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.409816 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.414096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.414937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.434623 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.434921 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:32.461403 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:32.498584 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:32.687547 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:01:32.709825 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:32.728436 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:32.740470 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.747439 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.753868 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.758733 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.774588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.775089 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.778210 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.778559 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.792910 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:32.795567 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.797823 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:32.807206 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:32.827592 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:32.932684 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.935974 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.938333 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:32.943953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:32.950109 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:32.950181 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:32.985627 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:33.047593 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:33.096398 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.122587 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.125084 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.133570 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.134076 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:33.134959 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.147748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:33.175596 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.246659 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.247051 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.247728 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.256984 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:33.260598 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:33.274879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.356948 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.357504 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.365339 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:33.368333 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.378612 29574 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:33.378744 29574 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:33.379168 29574 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:33.380988 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:33.382175 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:33.382251 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:33.382520 29574 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:33.383384 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:33.383817 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.413491 28156 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:33.413627 28156 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:33.414158 28156 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:33.415241 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:33.415582 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:33.415653 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:33.415802 28156 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:33.418397 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:33.466536 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:01:33.476104 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.478020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.480248 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.482115 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:33.488776 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 02:01:33.489002 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:33.490849 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:33.590711 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:33.640489 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:33.650417 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.676105 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.680640 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.683259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.683321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.685360 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:33.688091 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240503 02:01:33.692615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:33.740896 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.792925 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.795497 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:33.804342 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.804857 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.810142 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.812000 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:33.812378 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:01:33.819799 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:01:33.840673 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.840791 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:33.849964 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:33.855520 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:33.866662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:33.870371 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:33.880996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:33.889971 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:33.916440 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:33.971832 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:33.992341 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.160858 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.161347 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.164461 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:34.171407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.176522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.177642 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:34.225323 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.232959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.238749 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.239118 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.240653 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.244131 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:34.250691 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:34.257793 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:34.264954 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:34.313473 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.314299 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.317831 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:34.326706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.327306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:34.335129 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.359632 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:34.412851 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:34.452582 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:34.479728 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.560145 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.562510 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:34.566905 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.567255 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.576392 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:34.578596 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.581470 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.660997 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.666924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.667147 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:34.667568 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:34.670475 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.670601 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.714033 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:01:34.723445 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.769361 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.770768 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.779927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.783921 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:34.786110 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:34.797978 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.801223 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.808399 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:34.877688 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.884989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.887212 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:34.887324 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:34.889752 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:34.905048 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:01:34.906419 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:34.906648 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:34.907871 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.091037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.091245 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.094972 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.103367 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.109630 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.114147 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.219170 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.222399 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.228034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.235298 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.235570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.235780 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.235813 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.243319 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.248999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.256011 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.257652 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:35.289072 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:35.294586 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:35.326403 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:35.395377 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:35.412752 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:35.469415 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:35.572860 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.576900 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.578122 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.580668 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.581907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:35.601073 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:35.615859 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.640635 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.648135 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.648873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.650856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.652963 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.658881 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.751003 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.751327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.753738 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.756204 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:35.758690 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:35.765585 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:35.776722 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.907070 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.912719 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.914513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:35.914757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:35.922082 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:35.935797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:36.006883 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:36.030194 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:36.070991 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.072393 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:36.073246 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:36.074501 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.075451 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:36.079874 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:36.091823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:36.097978 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:36.180158 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.181252 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.193742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.195848 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:36.198397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:36.198813 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:36.254999 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.265061 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:36.265455 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.268151 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.272209 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.279520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.281915 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:36.292796 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:36.301204 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.322786 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.360987 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.425294 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.453033 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:36.485178 28679 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:36.485321 28679 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:36.485791 28679 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:36.486819 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:36.489204 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:36.489290 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:36.489437 28679 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:36.505280 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.508898 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.509472 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:36.509932 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.515697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.519973 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:36.602632 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:36.620862 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.623878 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.630473 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:36.630542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.633227 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.639261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.641364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.650009 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:36.658205 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.660063 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.684506 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:36.686779 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:36.835621 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:36.964308 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:36.983076 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.986124 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:36.997385 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.997448 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:36.999197 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.026863 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.026888 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.030607 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:37.030973 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:37.045634 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:37.054471 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.054555 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:37.064015 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:37.069634 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.137759 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.138679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:37.138897 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:37.139770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.140302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.141265 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.158520 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:37.174698 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:37.210871 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:37.244474 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:37.246681 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:37.271332 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:37.356273 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.359530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.362618 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.365533 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:37.371016 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.377656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:37.385414 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:01:37.438305 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:01:37.460906 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.461422 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.464864 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.468926 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.477533 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.484714 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:37.485891 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:37.561187 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.566737 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.568598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.576998 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.579349 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:37.579936 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:37.614830 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:37.689476 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:37.690821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.692015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.692399 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.693013 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.693887 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:37.723217 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:37.814314 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:37.870735 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:37.875959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.878126 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.881103 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.887619 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:37.887771 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:37.892328 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:37.953301 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:37.986135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:37.990413 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.003927 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.004297 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:38.013200 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:38.013660 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:38.022733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.055615 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.059674 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.059919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.060524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.060779 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.073807 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:38.260344 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:38.345908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.346729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.351526 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.365500 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:38.368316 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.371910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.432499 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:38.436902 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:38.442657 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.444149 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.445628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.447162 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.460434 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:38.483373 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:01:38.514271 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.516669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.522606 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.523967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.532572 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:38.534160 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:01:38.540401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:38.551533 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.563223 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.619108 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:38.729843 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.730227 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.737303 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:01:38.737890 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.749461 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:38.756937 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.757977 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.758944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.763418 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.767889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.769443 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:38.816855 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:38.823395 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:38.824100 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:38.855791 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.859676 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.863512 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.866086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.874567 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.876384 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:38.958842 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.965785 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:38.966413 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:38.967857 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:38.969025 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:38.969106 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:39.033723 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 02:01:39.034286 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:39.058171 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:39.074502 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.076576 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.078445 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.081933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.090607 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.111357 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:39.244521 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:39.273093 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.276485 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.277055 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.279073 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:39.279685 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.291906 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.391650 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.395135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.397580 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.412050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.412532 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:39.427114 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:39.431538 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:39.444883 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:39.446290 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.450873 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.452206 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.453125 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:39.460479 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:39.473380 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.485633 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:39.574483 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.595446 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.601891 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.603250 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.625144 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.684469 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.722407 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:39.767066 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.768431 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.769608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.780082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.782580 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.785146 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:39.794426 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:01:39.820154 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:01:39.828188 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.833050 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.835932 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.838749 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.839973 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.858420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:39.931274 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.931895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.934847 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:39.946424 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:39.948307 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:39.950837 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:40.000454 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:40.035547 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:40.105446 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:40.112612 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:40.151101 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.152209 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.152724 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:40.153506 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.164109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.183079 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:40.247037 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:40.263026 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:40.263257 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.263528 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.263499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.267304 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.280267 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.369786 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.369786 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.372725 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.379873 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:40.389384 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:40.396175 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.396562 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:40.402581 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:40.473593 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.477031 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.481364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.482007 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.490115 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.497737 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:40.509392 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:40.686717 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.688149 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.688634 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.690173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.690876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.704351 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:40.804735 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:40.816669 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:40.818296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:40.822779 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.830370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.834075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.836365 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.851979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.868598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.869235 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:40.869576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:40.872812 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:40.876467 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:40.912212 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:40.924996 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:40.939805 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:41.000589 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:41.165529 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:41.172724 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:41.184159 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.187086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.195256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:41.196205 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:41.196842 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:41.202809 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.206077 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.218863 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:41.221333 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:41.227916 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.247143 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.250108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.252528 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:41.268914 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:41.331019 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.333621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.334167 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:41.334928 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.338747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.357138 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:41.391165 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:41.402184 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:41.487313 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:41.555127 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.555949 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.557909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:41.558562 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.559688 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.584439 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:41.644810 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:41.650310 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.652608 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.665719 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.666759 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:41.666909 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.668115 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:41.709152 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:41.758252 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.767530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.767791 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.769732 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:41.775892 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:41.779234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:41.800405 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:41.866452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.869217 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.869510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:41.871043 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:41.874361 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:41.879848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:41.920846 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:41.932164 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:41.975476 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:42.069314 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:42.079844 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.080884 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.083350 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.088326 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.095242 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:42.101876 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:42.103864 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:42.109158 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:42.117869 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:42.199695 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.203125 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.212586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.215080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.220664 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.224296 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:42.249573 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.249855 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.251004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.266649 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.269367 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:42.270232 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:42.310988 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:42.321211 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:42.326061 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:42.353772 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:01:42.467386 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:01:42.546882 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.549515 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.554642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.558849 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.561434 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:42.562999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.602594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.610669 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.617831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.619217 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.623888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:42.632956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:42.694116 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:42.698169 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:42.707432 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:42.720840 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.722434 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.724443 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.727830 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.745175 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.757889 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:42.782276 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:42.790558 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:42.893394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.894407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.896441 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:42.897938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:42.901633 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:42.910239 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:42.912487 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:43.043417 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.047500 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.047500 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.048040 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.056656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.069918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:43.144109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.149561 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.150468 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:43.154870 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.159926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.170441 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.247663 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.250242 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.252635 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.253273 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.259919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.259932 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:43.277158 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:43.400097 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:43.413154 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:43.428681 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:43.450083 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:43.461709 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:43.464955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.466212 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.466252 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.476276 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.477308 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:43.493675 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:43.496642 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:43.517951 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:01:43.589700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.590770 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.594276 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:43.601876 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:01:43.603042 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:43.604862 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:43.612035 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.620877 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:43.636670 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:43.650701 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.653985 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.654394 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.657717 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:43.662325 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.676537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:43.746915 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 02:01:43.850375 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:43.850764 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:43.914988 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.915438 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:43.919721 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.930207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:43.931617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:43.940075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:43.962170 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.007828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.008113 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.010686 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:44.012501 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:44.013085 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.018184 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.024040 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:44.036407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:44.059412 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:01:44.117646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.118113 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.119676 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:44.126701 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.127571 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.130393 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:44.163537 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:44.239933 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.246475 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.247773 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.248075 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.256073 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.256275 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.258919 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:44.303057 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:44.399852 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:44.421191 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.425493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.426190 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.435072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.437088 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:44.442334 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.526309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.527004 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:44.529963 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.531632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.534291 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:44.544847 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.546577 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.550168 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.609238 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.609529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.612560 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.613322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.615585 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.617452 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:44.690091 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:44.713502 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:44.757905 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:44.797896 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:44.850068 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.854801 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.859272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.859982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.867986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:44.868325 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:44.902645 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.958313 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.959798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:44.963624 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:44.964131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:44.964819 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:44.967221 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:44.969810 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:44.974942 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:01:45.017355 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.023783 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.029459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.029901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.030366 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.032764 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:45.054534 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:45.067529 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:01:45.112951 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:45.155354 420 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:45.155494 420 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:45.155951 420 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:45.157001 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:45.157078 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:45.157132 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:45.157292 420 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:45.197376 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:45.252907 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:45.260303 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:45.285842 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.287532 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.292791 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.301057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.310564 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.325038 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:45.333863 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:45.374478 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.376135 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.382946 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.390602 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:45.396096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.406513 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.490671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.491410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.492216 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.492470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.504971 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.516326 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:45.614159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.619954 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.620003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:45.620400 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.620735 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.628901 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:45.650787 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:45.783389 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:45.799654 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.805972 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.807157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:45.808113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.808708 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.830076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:45.904429 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:45.914536 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.919692 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.930123 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:45.930689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:45.932366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:45.940335 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:45.951591 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:45.954880 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:45.978734 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.984364 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:45.989578 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:45.992486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.005952 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:46.015583 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:46.053138 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:46.086020 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:46.097795 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:46.194799 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:46.222538 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:46.253119 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:46.275585 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.279367 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.282215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.285195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.293424 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:46.298099 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:46.357748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.359395 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.373831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.379593 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:46.390331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.391742 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:46.414207 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:46.421739 479 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:46.421865 479 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:46.422303 479 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:46.424018 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:46.424165 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:46.424238 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:46.424477 479 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:46.443030 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.446254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.448058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.449122 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:46.456818 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:46.458967 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:46.548200 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:46.551911 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:46.618546 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:46.630020 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:46.690351 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.695478 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.697232 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.703964 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:46.704658 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.705432 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:46.791391 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:46.799890 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.801357 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.805672 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.811931 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.828230 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:46.834939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:46.909224 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.909336 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.910763 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:46.916728 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:46.917212 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:46.923710 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:46.941359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.008803 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.011721 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.013634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.021340 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.022475 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.040032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:47.052668 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:47.090371 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:47.201951 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:47.237715 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:47.238960 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:47.243463 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.244156 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.249737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.252264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.259552 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:47.330720 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.334707 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.336992 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.339744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.339891 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:47.351258 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:47.364869 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:47.388784 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.392292 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:47.401953 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:01:47.413457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.413846 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.416198 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.418000 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:47.425333 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:47.459069 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:47.485276 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:47.506017 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:47.535146 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:01:47.611371 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:47.638540 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:47.708062 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.710991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:47.713131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.713837 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.714514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:47.760478 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:47.767369 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:47.768795 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.774374 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.782363 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.795073 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.796171 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.811458 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:47.822784 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:47.891829 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:47.892174 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.895885 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:47.900197 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:47.904805 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:47.908114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:48.052691 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:48.094122 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.094431 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.097658 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:48.104564 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:48.105177 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:48.106952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.110242 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:48.204670 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.206593 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.208304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:48.219281 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:48.222784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.222900 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:48.239493 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:48.273356 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:48.307058 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.307298 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.308813 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:48.312490 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.313521 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:48.335822 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:48.413462 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:48.426126 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.427150 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.429157 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:48.431591 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.434827 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:48.450937 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:48.543706 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 02:01:48.603087 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:48.607039 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:01:48.608146 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:48.633687 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.635320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:48.638530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.646723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:48.654202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:48.655031 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:48.670431 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:48.674700 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:48.741330 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:48.774987 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.775846 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.779347 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:48.782179 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.790453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:48.791687 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240503 02:01:48.804101 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:48.838719 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.850553 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:48.850729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:48.869271 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:48.870007 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:48.880508 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:48.887084 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:48.967162 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:49.070844 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:49.086654 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:01:49.087237 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:49.172390 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:01:49.172644 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:49.202091 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.204603 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.220432 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.230361 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.231915 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:49.233664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.235296 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.237100 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.238690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.249006 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.266592 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:49.268143 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:49.286597 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:49.353310 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.354074 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.365794 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.366250 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.370496 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:49.377036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:49.454120 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:49.489252 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:49.566588 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.568506 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.573197 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.575677 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:49.576640 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.577076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:49.598959 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:49.632459 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:49.672305 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.675173 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.678777 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:49.680974 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:49.684075 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.689493 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.707089 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:49.730481 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:49.731402 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:49.739668 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:49.758502 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:49.793929 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.796154 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.798265 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.800382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.800967 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.803395 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:49.829668 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:49.897246 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.899382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:49.900023 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:49.901057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:49.917876 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:49.928273 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:49.962427 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:50.025535 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:50.072350 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:50.106178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:50.109822 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.110056 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.110448 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.111032 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:50.113134 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:01:50.119279 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:50.229300 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:50.237282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.241778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.242543 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.247236 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.249564 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:50.256407 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:50.260147 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.263392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:50.269531 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.274788 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:50.286226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.286767 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:50.294515 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:50.365074 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:50.584942 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.589363 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.590433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:50.596966 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:50.599697 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:50.614372 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:50.625849 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:50.637261 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:50.643626 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:50.644610 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.647826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.648938 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.649152 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.656505 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:50.683971 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:50.697165 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:50.726197 29934 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:01:50.726321 29934 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:01:50.726790 29934 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:01:50.727988 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:01:50.729326 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:50.729399 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:01:50.729526 29934 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:01:50.764168 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.764286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.767026 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:50.771950 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.778609 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.784572 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:50.860314 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:50.935562 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.935745 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.941398 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:50.942488 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:50.943989 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:50.951411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.085341 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.086133 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.087834 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.096118 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:51.097698 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.114567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:51.159464 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:51.200176 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.200477 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.203943 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.205731 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.208845 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:51.222250 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:51.234042 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:51.282658 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.284894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.291441 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.297982 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.299399 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.311434 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:51.336508 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.365433 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:51.440238 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.452916 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.465008 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.470664 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.471597 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:51.522533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.523679 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.527971 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.529711 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:51.536607 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.544276 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.634354 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.636638 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.640466 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.654132 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.656916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.661523 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:51.673051 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.688894 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:51.692492 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.702302 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:51.708456 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.719835 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:51.745600 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:51.806403 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:51.946273 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.956475 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.973915 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:51.982188 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:51.994921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:51.995560 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:51.997464 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.015022 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.016018 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.018211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.050578 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.051939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.070559 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.071168 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.072903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.085937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:52.151502 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:52.171692 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.177160 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.182636 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.189668 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.191108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.200147 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:52.228678 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:52.237493 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:52.342830 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:52.350100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.353487 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.360548 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.361018 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.363839 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.367480 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:52.456820 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:52.483906 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:52.484663 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.486609 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.487479 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.489764 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.506237 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.508620 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.588717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.589649 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.594974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.595986 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:52.607816 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:52.610630 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.622366 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:52.629007 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:52.677791 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:01:52.693697 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.696635 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.696642 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.699966 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.703997 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:52.707831 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:52.708972 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:52.737056 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:52.913878 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:52.917721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:52.919358 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.923139 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:52.937048 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:52.958587 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:52.962201 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.018347 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.023072 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.024297 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.024526 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:53.034289 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.036489 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:53.050163 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:53.100602 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:53.104379 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.106550 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.107268 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:53.110935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.118729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:53.119952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:53.125191 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:53.134008 2301 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:53.134135 2301 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:01:53.134522 2301 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:01:53.135469 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:53.140025 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:53.140096 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:53.140327 2301 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:53.239250 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:53.342084 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.342644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.345381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.349277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:53.359181 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:53.363194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.367218 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:53.455020 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.458314 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.458772 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.460968 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.462720 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:53.462903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:53.510098 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:53.533697 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:53.564148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.564838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.567807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.568660 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:53.577055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.601610 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:53.632490 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:01:53.684092 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.687453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:53.687719 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.688467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.690462 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.708194 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:53.779467 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:01:53.803669 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:53.831806 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:53.837170 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:53.865792 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:53.879532 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:53.890092 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.892897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:53.894369 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.894495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:53.904719 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:53.915421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:53.927743 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:01:53.988926 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:54.008277 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:54.016012 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:54.016719 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.024140 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.024788 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.049772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.050524 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.054757 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.055038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.060907 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.061616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.074749 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:54.086971 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:01:54.088111 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:54.100257 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:54.192281 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:01:54.193071 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:54.353853 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:54.356309 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:54.377631 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:54.392210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.392263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:54.404791 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:54.405066 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.407938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.408607 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.421713 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:54.431252 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:54.440291 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:54.441787 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.445097 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.446098 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.456524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.464550 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:54.474682 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:54.508816 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:54.544013 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.544055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.546283 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:54.547163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.556396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.558456 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:54.586072 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:54.780758 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.782518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.783223 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.788079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.793962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.808284 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:54.885378 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.885655 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.886368 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:54.897519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.898681 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:54.900491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:54.935021 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:54.935819 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:54.995066 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:54.995841 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:54.999137 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:54.999328 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:54.999723 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.000053 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.008749 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:55.067840 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:55.105614 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.108443 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.108646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.109493 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.117595 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:55.119189 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:55.133857 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:55.169569 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:55.174973 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:55.232978 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:55.310745 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.310871 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.313561 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.314025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.320560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.328420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:55.421100 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:55.439199 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.442674 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.445772 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.452369 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.456198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.459224 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.462010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.474025 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:55.476138 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.492596 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:55.501305 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.515312 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:55.556229 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:55.618958 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:55.749094 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:55.814776 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.814810 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.819936 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:55.825224 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:55.825904 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:55.827386 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.829447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.850689 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.858723 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:55.861153 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.868055 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.869458 3866 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:01:55.869562 3866 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:01:55.869967 3866 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:01:55.871836 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:55.872045 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:55.872098 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:55.872354 3866 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:55.877691 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:55.878482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.880988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.962450 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.963790 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.968513 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:55.970084 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:55.983902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:55.992723 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:56.009678 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:56.010476 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:56.054097 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:56.079617 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:56.177592 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.180256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.182199 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:56.196141 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.197511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.206748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.280934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.281325 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.284233 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.286343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.305467 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.305972 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:56.366518 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:56.368209 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:56.387346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.388669 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.392388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.395341 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.421636 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.422178 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:56.424911 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:56.499047 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:56.500556 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.501933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.506101 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.512701 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.516273 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.519776 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:56.522406 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:56.571482 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:56.608130 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:56.713330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.719702 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:56.721616 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.723830 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:56.732265 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:56.732892 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.743921 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.858409 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.862668 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.865890 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.866505 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.867771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.869230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:56.869503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.876259 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:56.878252 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:56.900287 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:56.901242 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:56.905561 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:56.912889 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:56.964236 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:57.025708 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:57.088469 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:57.089450 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:57.129287 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:57.205492 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:57.237988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.238253 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.238816 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.246969 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.247282 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.251957 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.262398 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:57.263916 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.264658 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:57.270992 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:57.272611 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:57.296444 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:57.350106 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:57.384451 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.386492 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.388000 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:57.388701 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.392870 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:57.400501 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:57.408358 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:57.420979 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:57.465086 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:57.528117 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:57.546725 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:57.583292 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.587333 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.589362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.590348 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.591251 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:57.591944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:57.660900 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:57.703611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.707794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.707825 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.710435 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:57.719623 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:57.725137 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.726459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:57.775338 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:57.808574 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.814960 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.815503 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:57.816010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.818249 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.821899 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:57.906575 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.910401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:57.912187 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:57.912825 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:57.924429 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:57.927426 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:57.928841 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:57.968508 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:58.126129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.126570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.129485 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.130303 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.132787 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.133528 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.244773 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.251356 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.257409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.263911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.269196 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.275913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.279393 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.282058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.292907 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.306398 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.308794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.311543 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:58.314836 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:58.331979 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:58.396596 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:58.398602 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:58.457033 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:58.554715 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:58.620182 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.627151 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.628141 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.632151 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.632448 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.632601 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.635062 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.638713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.657115 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.662382 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.665817 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.676237 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:58.724380 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:01:58.725008 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:58.767020 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.770840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.773156 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:58.776054 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:58.779040 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:01:58.788940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:58.796327 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:01:58.800025 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:58.908829 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:58.947330 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:01:58.975255 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.976528 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:58.978730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:58.982162 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:58.985879 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:58.986464 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:58.988003 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:59.010422 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:59.040513 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:59.082818 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.083148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.091358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:59.092880 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:59.096143 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.099086 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:59.183190 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:01:59.184012 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:59.197324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.199556 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.203147 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:59.210072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.211597 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:59.217089 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:59.299273 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:59.300277 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.300894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.301805 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.319891 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:59.320303 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:59.322034 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:01:59.330112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:01:59.361102 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:59.415575 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:59.434423 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:59.503450 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.504009 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.510211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.519997 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:59.525060 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:59.533320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:01:59.550148 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:59.557372 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:59.620607 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.626832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:59.631415 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:59.641861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:01:59.657846 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.662633 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.671963 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.677673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:01:59.678263 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:01:59.683496 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:59.684269 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:01:59.687973 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.699926 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:01:59.711012 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:59.789422 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:01:59.893558 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:59.924310 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:59.951779 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:59.988202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.993196 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.995896 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:01:59.999929 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:00.007616 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:00.009991 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:00.016991 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.041370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.044421 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.052328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.052843 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.056736 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.056928 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:00.072008 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:02:00.147682 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.149704 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.150336 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.159478 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.161343 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:00.163051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:00.173822 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:02:00.175166 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:02:00.202684 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:00.322052 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:00.368117 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.371191 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.373378 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.381983 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.383425 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:00.390640 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:00.408329 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:00.476037 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.476899 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.480269 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.484704 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:00.490379 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.502120 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.563741 5154 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:00.563907 5154 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:00.564342 5154 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:02:00.565064 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:02:00.566049 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:00.566121 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:00.566272 5154 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:00.584903 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.587534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.588901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.589870 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.597548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.598151 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:00.687269 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:00.687868 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.691354 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.691870 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.695915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.696151 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:00.698957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:00.842062 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:00.858248 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:00.888041 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:00.889816 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.891778 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:00.892261 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:00.894840 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.904070 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:00.916563 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:00.945686 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:01.012423 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.013193 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.015484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.016728 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.023917 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.034384 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.046736 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.056113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.056479 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.075961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.080566 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:01.088302 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:01.161818 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:01.220875 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:01.253150 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:01.296216 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:01.308796 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:01.322110 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:01.354517 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:01.383899 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.386762 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.394225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.408402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.412186 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:01.428376 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.435127 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.436265 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.436748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.439229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.472638 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.473244 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:01.507436 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:01.550149 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.551946 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.552749 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.558012 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:01.567974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.571220 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:01.657681 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:01.721606 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.724571 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.725050 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.729080 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:01.733866 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.752297 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:01.793735 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:01.844273 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:01.858863 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.861001 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.866851 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.868115 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:01.872488 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.894686 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:01.895545 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:01.908942 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:01.987866 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:01.988006 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.992292 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:01.993851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.004819 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.005112 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:02.036651 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:02.075110 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.078831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.082679 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.086107 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.086529 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:02.087909 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:02.107267 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:02.170536 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:02.215313 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:02.220984 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:02.328061 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:02.331905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.331926 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.338637 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.348153 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.356978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.357137 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:02.382232 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:02.395182 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:02.401566 4739 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:02.401674 4739 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:02:02.402026 4739 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:02:02.403033 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:02.403110 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:02.403173 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:02.403301 4739 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:02.435142 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.437587 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.438431 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.442121 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.466575 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:02.467063 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:02.474201 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:02.499141 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.502656 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.503506 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:02.511279 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:02:02.527241 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:02.527415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.530182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:02.561363 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:02.633340 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:02:02.698539 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:02.774533 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:02.791298 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.795146 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.795652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.805503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:02.808630 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:02.818006 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.828878 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:02.873684 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.876183 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.877357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.879920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.892060 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:02.892838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.979810 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.981062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:02.984020 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:02.985692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:02.986306 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:02.992568 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.116111 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.118716 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.126075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:03.128265 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.129495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.151069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:03.264643 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:03.284716 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.284832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.285517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.291843 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.297060 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:03.299732 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:03.306453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:03.360585 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:03.394374 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.395658 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:03.399140 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.402540 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.409143 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:03.409303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.484433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.488068 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.492151 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.504650 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:03.507790 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:03.508066 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:03.521998 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:03.523512 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:03.712823 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.716149 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.716198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:03.717792 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.719770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.733798 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:03.792584 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:03.814164 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:02:03.815069 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:03.833559 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.833969 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.840127 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.840803 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:03.844673 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:03.862296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:03.866876 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:03.887683 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:03.901392 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.901477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:03.902738 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.904086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:03.909150 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:03.909679 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:03.930593 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:02:03.931232 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:03.959133 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:02:03.987921 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:04.060199 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:04.067181 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:04.170624 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.170732 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.173693 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.179610 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:04.183130 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.192266 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:04.196293 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:04.257108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.257544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.258033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.269197 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.270606 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.284111 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:04.285452 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:02:04.290580 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:04.314548 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:04.366957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.372069 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.373301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.374241 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.378746 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.379583 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:04.384902 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:04.399814 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:02:04.491313 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.492765 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.499092 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.505627 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.512161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.512571 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:04.520220 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:04.543658 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:04.579139 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:04.681444 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.682696 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.687599 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.692479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.694389 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.701923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:04.804100 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.805878 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.808802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.809150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.809607 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:04.829739 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:04.830286 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:04.830672 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:04.831131 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:04.842784 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:04.843641 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:04.872673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:04.876658 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:04.877230 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:04.883911 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:04.884256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:04.884505 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:04.905520 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:05.015779 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:05.058344 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:02:05.130511 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.130789 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.133239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.134969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.141400 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.158542 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.231968 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.238072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.240151 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.240365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.250519 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.257193 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:05.275633 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:05.277709 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:05.286211 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.297957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.300565 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.302768 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.312742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.312901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:05.320396 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:02:05.338622 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:05.357101 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:05.580178 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.588167 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.588814 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.595952 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.596199 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.608392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.657294 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.658690 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.659902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.663973 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.672382 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.679920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.761049 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.767485 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.769791 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:05.770133 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:05.774505 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.793457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:05.793938 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:05.835161 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:05.897876 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.900708 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.905632 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:05.907825 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:05.915596 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:05.922588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:05.924566 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:05.953956 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:06.085906 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.090417 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:06.092573 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.092823 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:06.094236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.112331 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:06.149786 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:06.149861 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:06.202419 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.203245 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.204134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.208793 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:06.218258 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:06.234642 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:06.235285 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:06.273345 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.274363 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:06.274919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.278404 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:06.279651 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.282469 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:06.354410 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:06.479404 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:06.500030 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:06.524188 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.525403 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.535691 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.536087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:06.536413 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:06.536422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:06.537164 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:06.549746 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:06.632790 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:06.642305 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.648939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.651047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.655649 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:06.656257 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:06.656419 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:06.675406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:06.677908 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:06.679769 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.687647 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.697814 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:06.713148 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:06.733506 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:06.797477 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:06.861925 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:06.903167 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:06.931532 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.012805 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.017599 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.021629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.027086 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:07.036671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.038493 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.039212 6269 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:07.039312 6269 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:07.039791 6269 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:02:07.040902 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:07.045333 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:02:07.045414 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:07.045629 6269 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:07.052722 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.078608 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.082944 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.085141 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.086347 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:07.092558 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.094839 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:07.156507 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:07.181221 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.186767 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.189962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:07.190521 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.192133 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:07.205494 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.206130 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.212054 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:07.273366 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.340378 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.341943 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.343430 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.351006 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.362747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.364172 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:07.400522 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:07.468498 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:07.475608 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:07.495070 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.498229 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.505088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.505095 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.513159 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:07.520305 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:07.522209 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:07.588537 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:07.601064 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:07.603268 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.603360 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.604650 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.605381 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.607209 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.626418 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:07.637238 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:02:07.665158 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:07.698251 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:02:07.709338 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.709924 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.711855 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.718463 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.720715 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.737669 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:07.897784 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:07.913631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.914053 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.914677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:07.920274 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:07.936486 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:07.938874 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:08.042028 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:08.042297 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.044467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.048130 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.060154 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.064628 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.070297 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.076130 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.102759 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.103269 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.105381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.105752 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:08.188738 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:08.406560 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:08.419850 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.421813 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.430361 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.430586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.445032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.445166 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:08.465445 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.467311 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.474200 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.476776 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.492470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.495934 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:08.521237 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:08.552198 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:08.578533 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.579188 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.580567 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.581044 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.587688 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.600970 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:08.647297 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:08.695223 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:08.802778 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.805079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:08.806425 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.807394 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.811988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.816709 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:08.818413 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:02:08.879249 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:08.887003 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:08.899185 7345 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:08.899320 7345 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:08.899776 7345 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:08.902261 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:08.904961 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:08.905315 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:08.905565 7345 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:08.906826 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.908087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.908361 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:08.908813 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:08.910506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:08.920984 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:08.941943 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:02:08.942587 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:08.954146 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:08.974673 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:08.993489 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:09.008895 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:02:09.009131 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.009210 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.011490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.019902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.021994 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.028919 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:09.074373 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:09.120244 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.120766 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.123798 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.124580 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.125226 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.143992 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:09.270874 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:09.318985 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.324445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.325040 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.325131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.326638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:09.335364 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:02:09.344038 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:09.364318 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:09.425307 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:09.451067 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.453301 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.462723 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:09.462846 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:09.464797 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.464890 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:09.465173 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.473388 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.474046 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.484745 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.495304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.497749 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:09.518868 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:09.549317 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:09.549896 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:09.678505 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:09.678496 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:09.679193 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:09.757907 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:09.825621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.830308 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.830437 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.835038 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.841457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.842482 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:09.842832 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:09.868942 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.869684 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:09.870011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:09.877434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.878397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:09.886325 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:09.888774 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:09.932569 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:09.946130 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:09.981696 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.982398 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.983359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:09.984874 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:09.987344 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.005439 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:10.007973 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:10.123528 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:10.137037 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:10.166574 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:02:10.199355 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:10.202443 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.206431 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.206688 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.209229 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.215224 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.215881 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:10.308403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.309844 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.311316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.321635 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.321792 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.328279 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:10.369087 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:10.422302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.425938 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.429585 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.430073 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.433073 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.434456 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:10.451983 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:02:10.521106 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.521512 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.522403 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.527420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:10.527658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.534299 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:10.544526 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:10.715786 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:10.739323 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.740353 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.744683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.752110 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:10.752465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.775800 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.857964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.866405 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.866693 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.867316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.872081 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:10.876799 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:10.887920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:10.893292 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:10.902395 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.907923 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:10.912366 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.928726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:10.931308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:10.943679 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:10.968462 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:11.148236 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:11.205451 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:11.222877 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.229306 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.231037 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.231832 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:11.233685 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:11.235172 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.248003 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:11.265725 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:11.282815 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:11.295352 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.295841 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.301797 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.311667 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.311919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.313849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.406078 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.408982 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.411728 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.411929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.424235 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.428663 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:11.435822 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:11.457870 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:11.471343 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:11.539871 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.542644 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.546350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.553627 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:11.555189 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:11.558362 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.572176 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:11.689615 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:11.689983 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:11.717288 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.720782 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.722200 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.722887 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.730950 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.733886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.847326 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.849045 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:11.849325 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.850502 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.863139 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.864125 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:11.892525 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:11.906767 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.913822 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:11.914878 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:11.923435 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:11.924463 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:11.930815 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:11.936435 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:12.067734 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.183734 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.184376 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.193905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.197966 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:12.206980 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.209162 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:12.216202 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:12.252957 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:12.258270 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:12.285995 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.287451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.295189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.297217 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:12.301383 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:12.314689 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:12.318354 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.322535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:12.325385 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.326977 6269 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:12.327075 6269 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:12.327450 6269 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:02:12.328088 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:02:12.332624 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:12.332695 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:12.332937 6269 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:12.346221 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.356069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:12.356104 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.360687 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.360688 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.370447 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:12.375679 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:12.481968 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:12.495383 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:12.511425 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.659194 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.666596 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.670169 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:12.672642 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.672719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:12.688062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.727924 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.728163 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.734813 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.739121 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:12.742244 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:12.753741 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:12.822093 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:02:12.843679 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:02:12.850013 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.851114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.851729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:12.852607 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.852862 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:12.865204 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:12.890861 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:12.899545 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:12.937077 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:12.986183 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.988803 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.991865 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:12.994050 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.010823 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:13.011493 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.157630 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.159986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.163020 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.168254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.170109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.171797 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:13.270689 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.272981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:13.276575 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:13.277992 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.280236 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.282217 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.282640 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:13.349155 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.349967 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.355522 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.359930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:13.364308 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:13.366986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.367169 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:13.394201 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:13.427922 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:13.593271 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:13.594452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.594456 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.598881 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:13.605438 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.605718 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.627578 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:13.671087 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:13.711867 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.712705 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.723793 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.726994 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:13.729964 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.743466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:13.753250 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:13.774449 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:13.795522 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:13.796080 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:13.799651 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.799996 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.817481 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:13.818714 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:13.819090 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:13.819900 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:13.853273 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:13.887943 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:02:14.031304 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:02:14.058935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.062157 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.067943 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.071244 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.078282 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:14.083927 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:14.091022 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:02:14.091668 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:14.151871 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.152505 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.153852 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:14.154385 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.167716 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:14.180658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:14.220885 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:14.260838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.261220 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.261459 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:14.264971 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:14.265944 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.269372 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:14.286113 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:14.288831 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:14.357995 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:14.383955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.392243 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.392243 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.393754 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:14.396818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.397490 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:14.413092 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.495788 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:02:14.496480 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.507040 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:14.577330 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:14.577677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.577924 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.580407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.581163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:14.582257 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.683257 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.689296 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.689872 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:14.696381 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:14.698406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.709228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.746488 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:14.749706 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:14.765595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:14.768980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.770995 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:14.779059 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:14.781029 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:14.788162 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:14.800642 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.818140 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:14.834460 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.871068 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.891830 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:14.950052 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:14.966029 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:15.021904 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.029529 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:15.030041 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.037504 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.037768 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:15.047111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.119406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.120221 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.128952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.135192 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:15.153285 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:15.161634 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:15.168073 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.178226 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.188871 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.191169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.191679 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.195269 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:15.195961 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:15.317821 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:02:15.390055 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:15.484899 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.488464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.488474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.488765 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:15.490350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.497624 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:15.516045 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:15.551329 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.553769 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.560519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:15.562018 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:15.563503 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:15.569279 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:15.569869 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:15.623775 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:02:15.654242 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.658833 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.658874 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:15.660523 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.669039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.680742 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:15.703591 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:15.804724 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:15.806066 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:15.808095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.809324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.810453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.810799 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:15.818113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:15.962621 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:15.970392 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.974228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:15.974397 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:15.974663 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:15.986922 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.004930 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:16.082479 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:16.087494 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:16.093708 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.094972 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.097739 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:16.106987 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.107657 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.130236 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:16.169056 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:16.173243 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.179204 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.182550 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:16.190395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.192330 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:16.200975 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:16.212601 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:16.239321 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:16.317044 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:16.402673 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:16.420202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.422161 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.423887 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:16.433188 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:16.436234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.436976 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.528780 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.529042 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.534402 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:16.537102 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.549775 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:16.561326 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:16.573410 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:16.592790 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.593394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.602488 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.603927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.615437 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:16.617818 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:16.636827 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:16.642318 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:16.740535 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:16.866351 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.878823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.879310 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.883724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:16.885975 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.886241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:16.921483 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:16.948137 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:16.952852 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:16.954473 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:16.955868 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.957742 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.958757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:16.960153 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:16.973181 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:16.993216 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:17.061470 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.062922 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.063185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:17.065049 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:17.069366 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:17.072656 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.076937 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:17.136322 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:17.173928 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:17.194542 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.199199 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.202566 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:17.205363 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:02:17.205601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:17.215101 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.215560 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:17.313892 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:17.341532 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:17.377842 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.377985 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.380360 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.388785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.392364 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:17.399351 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:17.409397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:17.491032 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:17.497102 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.497740 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.498863 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:17.501742 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.509861 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.511384 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:17.559602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.561311 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.564733 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:17.565081 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.567720 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:17.568698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:17.643168 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:17.689468 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:17.781999 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:17.806432 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:17.822623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:17.828368 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.832655 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.833338 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.833994 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.844738 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:17.917986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:17.920181 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.924239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:17.925194 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.928440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:17.934643 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.009147 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.012640 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.014084 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.015947 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.021538 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.034116 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:18.048600 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:02:18.053289 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:18.072250 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:18.131561 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:18.188820 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:18.252430 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.254276 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.258013 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.258994 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.267508 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.269420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:18.307721 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:18.316680 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:18.360643 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.360991 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.365073 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.370254 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.370626 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.377597 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.473611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.476404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.484365 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.485558 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.487269 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.498250 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.583158 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.586211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.586669 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.587136 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.589156 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.607836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:18.654052 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:18.795804 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.797634 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.799898 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.800448 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.800937 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.808723 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:18.809321 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:18.811287 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:18.911479 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.920267 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:18.921316 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.923156 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.930761 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.936280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.970628 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.973054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:18.977092 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:18.978039 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:18.986034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:18.996953 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:18.999050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:19.077593 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.079118 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:02:19.079854 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.123219 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:02:19.123824 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:19.127602 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:02:19.148766 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.238673 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.241585 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.242080 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:19.246114 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.250847 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.252513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:19.265467 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.320727 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.322954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.323879 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:19.326014 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.336078 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.339689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:19.371351 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.373201 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.409860 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:19.445690 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.446115 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.452273 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.465334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.475126 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:19.475137 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:19.498889 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.558400 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.585798 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.590189 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.596992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.599388 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.614770 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:19.624732 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:19.714906 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:19.783690 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:02:19.786623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.786731 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:19.787304 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:19.787323 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:19.796521 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.796567 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:19.799401 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:19.803228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:19.806988 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:02:19.821858 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:19.831842 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:19.832401 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.903961 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.905552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.908579 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.916622 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:19.916741 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:19.921517 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:19.928311 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:19.967020 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.971576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:19.973059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.978571 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:19.985468 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.007292 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:20.294034 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.300094 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:20.303838 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:20.306421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:20.309314 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:20.311136 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.315913 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.324643 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:20.329071 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:20.349102 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.355894 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:20.359527 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.372006 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.374058 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:20.384902 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:20.388510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:20.404384 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:20.450017 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.455214 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.457562 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.460021 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.462277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:20.483606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:20.572636 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:20.654634 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:20.669997 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:20.673414 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:02:20.685374 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.686655 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.686794 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.688889 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:20.691570 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:20.708570 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:20.725844 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:20.745111 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:20.793370 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.797631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.798091 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.804491 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:20.805656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.816087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:20.868642 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:20.897125 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.900036 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.901551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:20.909018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:20.909485 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:20.925019 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.009029 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.010201 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.013702 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.018055 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.018529 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.041006 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.131050 10580 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:21.131179 10580 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:21.131589 10580 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:21.132781 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:21.132880 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:21.132942 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:21.133086 10580 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:21.145694 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:21.214212 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.214218 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.215327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.230033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.233614 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.248464 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:21.287186 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.337337 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:21.339563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.342476 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.350236 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.370177 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.370393 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.370687 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.379217 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.390717 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.392066 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.393373 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.394831 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.395445 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:21.463665 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.521795 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.552505 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:21.569221 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:21.641876 11104 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:21.642004 11104 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:02:21.642428 11104 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:02:21.643411 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:21.643492 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:21.643576 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:21.643821 11104 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:21.698580 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:21.728578 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.732180 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.733920 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.738540 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.749429 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.752272 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.756377 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.768122 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.774124 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.782768 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.786100 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:21.796334 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:21.822180 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.827711 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:21.831727 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.876593 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:21.890602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.894196 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:21.895385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:21.899250 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.899953 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:21.912385 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:21.923579 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.931489 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:21.995688 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:22.091637 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:22.095813 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.100893 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.104583 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.111351 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.112129 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:22.129359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:22.156926 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:22.177588 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:22.211217 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.212301 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:22.215729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:22.220358 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.222177 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.222877 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:22.247547 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:22.310864 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.314498 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.315311 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:22.315838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.318221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.335913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:22.425865 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.427875 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.429904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.430236 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:22.433620 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.443920 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:22.460657 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:22.589794 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:22.628403 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.629243 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.631902 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.632431 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:22.636445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:22.638397 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:22.639586 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:22.701714 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:22.747640 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.748602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.755939 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:22.769047 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:22.782522 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.788307 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.788866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:22.790277 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.796906 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:22.811245 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:22.815848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.824980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:22.847848 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:22.938895 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:23.038818 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:02:23.117110 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:23.122632 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:23.125217 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:23.126170 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:23.131966 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:23.133189 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:23.135926 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:23.136211 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.138244 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:23.149103 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:02:23.191795 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.198783 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:23.202584 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:23.203922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.204595 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.215246 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.221505 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:23.227702 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:23.283967 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.287119 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.287906 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.300086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:23.311908 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.312721 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:23.333076 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:23.513924 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.518308 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.519030 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:23.522272 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.525337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.550137 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:23.612462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.613520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.617659 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:23.624689 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.625229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.627197 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:23.639953 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:23.660275 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:23.712517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.714812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.720822 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.722666 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:23.733572 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.737186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:23.812049 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:23.844269 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.847083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:23.855482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:23.860937 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:23.863086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:23.869232 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:23.873499 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:23.946455 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:24.043145 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.044267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.051275 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:24.052480 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.052891 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.055914 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:24.119305 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:02:24.126093 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:02:24.126713 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.138969 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:02:24.153281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.157438 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:24.161089 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.163777 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:24.164026 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.167038 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:24.193830 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:24.216951 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:24.231873 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.233781 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.235487 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.236397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:24.240554 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:24.245163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:24.252627 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:24.262809 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.271456 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.273149 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:24.432201 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.446740 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:24.472170 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.475854 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.477116 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.479815 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.486093 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:24.498464 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:24.502453 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:24.571909 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.577816 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.579890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:24.583087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.587225 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.590570 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:24.596033 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:24.643930 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.645712 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.645989 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.648066 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.649901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:24.657931 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:24.700822 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.800609 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:24.838536 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:24.884924 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:24.889124 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:24.890898 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:02:24.897984 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.898054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:24.899433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.903412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:24.908399 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:24.912837 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.001858 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.006395 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.006836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.008740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.008756 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.032028 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:25.069188 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:25.113834 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.115145 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.115736 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.117692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.121420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.124547 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:25.204727 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:25.221771 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.222770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.227653 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.228412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.236511 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.239928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:25.313920 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:25.383365 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:25.413126 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:02:25.423167 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.425900 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.426466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:25.436621 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:25.442593 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.442868 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.443094 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:25.502142 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:25.540558 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.543977 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.548202 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.558254 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.560709 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.563567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.586218 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.588929 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.589264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.591486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.595209 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.601683 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:25.648344 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:25.691820 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:02:25.702847 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:25.853679 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:25.855105 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:25.915457 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.918154 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.928426 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.931634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.946482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.950897 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:25.951128 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:25.952665 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.954741 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:25.971024 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:25.972318 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:25.979358 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:25.985539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:26.021448 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:26.071280 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.073230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.076061 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.076536 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.099740 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.100386 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:26.284166 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.284341 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.288719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.297082 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:26.298023 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:26.298317 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.306416 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:26.308802 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:26.395495 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.395964 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.396890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.397899 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:26.413252 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:26.418514 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.422413 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:26.494004 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:26.504506 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.505029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.506841 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.516036 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.523355 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:26.524101 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.601910 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.603574 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.604001 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.606655 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.609265 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.610236 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:26.722558 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:26.817088 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.818151 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.819496 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.822356 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:26.822520 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:26.829664 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:26.831059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:26.834899 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:26.854524 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:26.884559 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:26.897432 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:26.935848 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.937755 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.941919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.950475 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.954545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.955682 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:26.962551 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:26.962611 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.971926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:26.978433 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:26.991005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:26.999539 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:27.000553 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:27.001272 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:27.034518 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:27.045387 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:27.103376 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:27.116847 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:27.283962 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:27.305392 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.306191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.308313 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.325057 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:27.328330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.332049 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:27.336897 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:27.340294 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.340802 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.342463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:27.351904 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:27.370368 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:27.384914 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:27.451723 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:27.470098 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.470484 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.474607 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:27.476693 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:27.477385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:27.486374 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:27.486794 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:02:27.549224 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:27.670073 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.672107 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.680961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:27.681633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.681816 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:27.692873 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:27.778366 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:27.786234 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.792671 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:27.795917 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.796726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.803468 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:27.803786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:27.889976 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:27.895325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.898149 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:27.899471 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:27.901240 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.907027 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:27.915668 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:27.957248 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:28.005774 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.006975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.011171 11938 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:28.011283 11938 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:28.011806 11938 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:02:28.012938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:28.012962 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:28.013583 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:02:28.013665 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:28.013902 11938 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:28.015365 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.015944 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.040606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:28.080670 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:28.120429 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:28.148577 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:28.215662 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.216066 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.219419 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.219568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.221401 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:28.235939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:28.283020 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:02:28.332165 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.333671 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:28.341565 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:28.343483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.344410 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:28.345198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.365195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:28.375687 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.380378 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.387025 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:28.404109 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.404922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:28.410521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:28.416405 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:28.504101 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:28.550276 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:28.694765 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:28.696837 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.698993 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.699636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.703914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:28.705523 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:28.705956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:28.711956 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:28.763515 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.770815 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.776710 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:28.777186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.783939 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.804410 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:28.858418 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.859889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:28.860386 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.861023 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:28.865729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:28.888151 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:28.973256 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:28.980906 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.086623 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:29.095237 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.100675 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:29.101588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.101850 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:29.102309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.116217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:29.161757 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:02:29.162171 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 02:02:29.184346 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:02:29.206225 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.207733 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.208047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.210362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.223455 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:29.230577 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:29.255388 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:29.310191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.315780 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.316488 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.326474 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:29.327533 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:29.329703 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:29.339244 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.350421 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:29.423410 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.424860 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.425472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.429322 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:29.429322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.446290 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:29.460762 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.522220 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.524287 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:29.526543 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.570422 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.580256 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:29.628721 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.629254 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.631978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.639362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.639958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:29.644299 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:29.656062 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:29.678073 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:29.678194 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:29.754326 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.756222 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.761952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.764305 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.788836 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.790796 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:29.792017 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.793579 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:29.801609 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:29.807631 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:29.810034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:29.832998 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:29.844578 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:29.902168 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:02:29.902765 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:30.018693 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:30.033121 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:30.033831 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:30.105713 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:30.136695 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.138657 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.146932 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.147462 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.152958 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:30.165369 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.169103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:30.173852 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.174343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.179208 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.192642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.204273 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.292408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.293653 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:30.297191 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:30.298883 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.300719 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.305904 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.312191 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:30.406190 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:30.493224 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.493378 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.496451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:30.504252 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:30.504482 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:30.507045 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.518152 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.518797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:30.580422 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:30.606069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.607565 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.612864 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.613253 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:30.613718 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.625734 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:30.662252 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:02:30.680924 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:30.682391 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:02:30.710978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.713074 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.713869 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.716943 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:30.722873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.749929 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.821959 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.824611 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:30.826602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:30.827955 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:30.830205 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:30.833596 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:30.934031 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:30.981089 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:31.009459 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:31.037124 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.039155 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.050850 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.051256 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.059068 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.070329 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:31.150128 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.151376 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.162846 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.168920 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.169082 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.172315 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:31.213855 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.220115 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.221392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.223739 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.226675 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.227038 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:31.334583 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:31.372445 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:31.486780 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.487798 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.490993 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.501333 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.506394 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.508989 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:31.513100 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:31.534449 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:31.570678 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.571413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.573875 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.576105 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.581557 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:31.603725 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:31.605757 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:31.676038 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:31.682643 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.685029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.693948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:31.696826 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.700472 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.711948 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:31.742270 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:31.810567 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.813026 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.816231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:31.819078 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:31.819459 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:31.819926 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:31.890528 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:31.947769 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:31.971978 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:32.000638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.001607 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.002596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.003207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.004213 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.005838 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:32.038620 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:32.113215 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:32.119470 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:32.120715 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:32.121580 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:32.127059 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.133033 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.141039 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.141494 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.155215 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.172708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.174155 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.183429 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.188490 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.196413 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.196707 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:32.262488 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:32.265090 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:32.345634 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:32.365666 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:02:32.405671 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:32.485455 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:32.499138 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.500324 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.503201 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.521294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.523149 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.537117 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.547936 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.550220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.562662 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:32.562882 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:32.566606 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:32.575238 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:32.576050 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.588285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.661451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.662201 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.663038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.665408 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.680739 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.687336 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.876242 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.880769 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:32.884948 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:32.889470 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:32.895298 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:32.903522 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:32.943560 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:33.000038 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:33.005268 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:33.010905 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.012305 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:33.017498 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:33.022866 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:33.023236 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:33.023859 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.035470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:33.038614 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:33.044284 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.123862 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.124718 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.130839 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:33.136266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.142666 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:33.157092 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:33.188838 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:33.229782 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:33.230468 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.231546 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.235975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.240276 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.248481 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:33.307994 15325 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:33.308108 15325 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:33.308449 15325 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:33.309321 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:33.309412 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:33.309491 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:33.309631 15325 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:33.341636 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:02:33.450058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.451606 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.454289 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.454408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.456537 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:33.470238 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:33.489209 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:33.557727 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.563617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:33.567912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.567962 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:33.570331 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:33.583858 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.589079 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:33.616482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.629570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.631078 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:33.635881 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.643080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.653762 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:33.657073 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:33.677153 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:33.728014 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:33.898845 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.901598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.913506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:33.914674 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:33.933650 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:33.936285 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:33.938975 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:33.944465 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:34.008224 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.009040 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.011479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.011876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.012382 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:34.021241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:34.069927 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.110589 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:34.113790 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.115097 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.116673 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.120668 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:34.122933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.134160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:34.176229 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:02:34.242898 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.242955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.245393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.245591 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.255522 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:34.257205 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:34.307159 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:02:34.351481 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:02:34.352136 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.415571 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:34.422312 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.422833 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.425945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.432718 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.435777 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.445767 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:34.519906 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:34.536475 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.538359 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:34.538535 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:34.540937 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:34.541154 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.544306 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.563858 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.607141 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.610394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:34.610661 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:34.612223 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.612629 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.613881 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.638248 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:34.640292 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.661424 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:34.667192 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.701398 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.802206 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:34.850185 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:34.864717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.865650 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.867359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:34.876209 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:34.876312 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.885219 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:34.914194 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:34.942091 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:34.982071 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.986040 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:34.992238 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:34.995012 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.004279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.009644 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.012002 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.014811 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.015520 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:35.030972 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.032058 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:35.038473 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:35.057366 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:35.085243 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:35.085863 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:35.148854 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:02:35.149417 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:35.272922 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:35.281828 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:35.364465 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.369246 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.369711 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.372064 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.372701 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.383044 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.386612 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.403936 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:35.410955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.428887 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.437142 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.445721 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:35.456430 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:35.463429 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:35.517730 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.518261 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.521023 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.532018 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.532390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.543871 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:35.550477 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:35.589859 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:02:35.620570 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:35.731534 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.736294 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.739331 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:35.740909 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.752331 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.775492 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:35.831426 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:02:35.841049 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.842463 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.845280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.846599 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:35.859931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.859952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.964015 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.967818 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:35.967928 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.968125 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:35.968410 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:35.989825 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:36.047804 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:36.070050 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:36.070350 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.073053 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.073204 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:36.083011 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.089421 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.091940 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:36.103257 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:36.274787 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.275131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.276770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.281173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:36.289029 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:36.290822 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:36.291518 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.396209 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.406404 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:36.408689 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.409216 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.426998 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.429469 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.431486 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:36.434413 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.440164 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.441394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:36.449760 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:36.455257 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:36.457399 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.473086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:36.500411 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:36.553893 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:36.672406 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:36.755924 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:36.792542 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.794081 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:36.805056 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:36.806535 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.807055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.812034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:36.814122 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:36.827919 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:36.828729 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:36.830579 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.839957 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:36.850148 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:36.852988 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:36.872262 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:36.879532 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:36.941054 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:36.962898 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.966300 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.968107 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:36.973474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:36.975867 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:36.976325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:37.056247 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:37.109010 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:37.144420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:37.144975 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.146234 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.147518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.147609 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.149117 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:37.195978 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:37.224143 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:37.270545 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.272792 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.272818 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.288352 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.296305 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:37.301012 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:37.347687 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:37.358081 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:37.378139 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.379992 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.391793 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.393442 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.395774 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.409152 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:37.469120 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:37.471079 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:37.491058 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:37.492146 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.496376 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.497413 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.501114 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:37.503999 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.504717 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:37.627854 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:37.697387 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.702929 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.703835 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.703913 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.706171 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.727743 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:37.776424 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:37.825554 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.826108 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.830845 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.843370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.847887 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:37.849181 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:37.850167 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.857069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:37.859493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.863364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:37.868763 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:37.876446 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:37.902359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:37.916257 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:37.932444 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:37.980480 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:38.027228 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:02:38.076733 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:38.181370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.185407 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:38.185384 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:38.186368 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:38.188989 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:38.204319 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.213734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.214821 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:38.229488 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:02:38.248945 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.254143 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.255842 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.259176 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:38.259879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.285881 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:38.363656 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.367767 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.371897 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:38.377921 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.380635 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:38.382302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:38.513332 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:38.549592 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.551272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:38.553218 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.554163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.555903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.576718 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:38.618870 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:38.667513 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:38.679147 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.679523 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.685740 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:38.688393 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:38.692488 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:38.701220 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.706001 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:38.713153 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:38.788803 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.790414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:38.792953 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.801986 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:38.804637 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.805056 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:38.845999 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:38.878886 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.886365 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.887159 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:38.887928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:38.888190 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:38.890429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:38.899981 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:39.104362 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:39.107242 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.109460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.109733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.116530 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:39.117364 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.118999 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.143018 17530 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:39.143137 17530 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:02:39.143484 17530 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:02:39.147053 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:39.147159 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:39.147229 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:39.147533 17530 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:39.150990 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:39.205348 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:02:39.220888 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.231057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.235096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.236503 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:39.252388 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:39.254844 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.261883 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.266525 16897 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:39.266628 16897 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:39.266978 16897 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:02:39.268126 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:02:39.268468 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:39.268549 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:39.268751 16897 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:39.268777 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.273036 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.275957 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:39.283066 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:39.290664 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.292171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.293547 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.294807 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:39.338989 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:02:39.460690 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:02:39.461365 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:39.495332 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:39.525847 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:39.592829 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.597406 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.600394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:39.600947 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.602437 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.603603 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.626590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.629112 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.644445 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.644994 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.655007 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:39.655690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:39.656800 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.664074 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.665484 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:39.724159 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:39.760499 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.761262 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.762817 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.765240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.777227 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.795746 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:39.839063 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:39.948951 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:39.972236 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:39.972239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.973713 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.976116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:39.982451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:39.993963 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.083948 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.084363 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.093508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:40.096400 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.099973 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.106148 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:40.112259 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:02:40.113112 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:40.118723 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:40.170480 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:40.174278 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:02:40.188983 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:40.189121 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.200829 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.206506 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.212421 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.213280 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:40.268198 15325 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:40.268314 15325 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:40.268775 15325 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:40.270417 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:40.271574 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:40.271661 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:40.271842 15325 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:40.288077 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.289893 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.294822 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:40.307864 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:40.307900 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.317004 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:40.352694 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:40.390376 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:02:40.402721 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:40.424557 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:40.485738 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:40.492146 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:40.508713 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:40.508951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.510915 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.514720 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.525480 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:40.527702 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:40.540395 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.613935 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.614893 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.616895 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.631547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:40.638553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:40.643970 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.668063 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.671104 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:40.672947 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.673615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.677656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:40.688064 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:40.689208 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:40.943702 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:40.990286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.993382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:40.994562 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:40.996977 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:40.997684 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:40.998344 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:40.999938 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:41.045004 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:02:41.058739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.060144 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.060410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.060858 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:41.062372 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:41.062497 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:41.090852 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:41.155648 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.156486 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.159893 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:41.163892 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:41.165544 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.166379 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:41.294392 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:41.312749 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.315680 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.316943 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:41.319806 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:41.322280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.323732 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:41.324316 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:41.387777 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:41.420660 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:41.471421 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.473083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.475636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:41.477159 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:41.485343 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:41.500294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.576795 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.577754 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.581073 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:41.590065 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:41.595055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.605686 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:41.608685 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:41.662808 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.664112 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.666638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.668865 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:41.676513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:41.676803 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:41.775166 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:41.839490 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:41.856926 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:41.861855 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:41.880482 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:41.883074 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:41.904508 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.908982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.910041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:41.911530 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:41.911540 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:41.911919 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:41.932750 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:42.004596 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:42.021884 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.030560 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.035311 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:42.037267 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.037256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.040109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:42.083348 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:42.093432 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.096510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.098735 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.099980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.108922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.120637 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:42.171581 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:42.267269 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:42.286469 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:42.318224 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:42.346926 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.350070 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.357359 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.361915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.375589 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:42.382944 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:42.384433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:42.390236 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:42.421980 15325 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:42.422104 15325 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:42.422554 15325 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:42.423563 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:42.423632 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:42.423681 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:42.423839 15325 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:42.450071 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.450675 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:42.450803 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.451614 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.460924 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.463927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.549733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.551005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.553987 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.557996 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:42.564093 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.565057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.675093 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.675127 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.677268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.683888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:42.684949 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:42.688092 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:42.719359 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:42.783089 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:42.791525 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:42.838693 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:42.850688 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:42.859100 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.860587 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.861974 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.867905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.872937 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:42.888296 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:42.889194 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:42.987515 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:42.995409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:42.999504 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:42.999727 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.000394 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.016230 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.016541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.023128 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.024960 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.028460 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.046840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:43.053457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.068794 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:43.179849 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:43.213768 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:43.231236 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:43.251413 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:02:43.367401 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.370520 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.378175 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.384109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:43.389583 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:43.391033 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.399580 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.414922 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.416225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.421134 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.426479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.427974 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.524911 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.525779 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.529772 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.540205 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.540664 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.551349 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:43.564610 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:43.658680 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:43.668148 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:02:43.698920 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:43.714622 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:43.727852 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.730438 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.731040 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.746119 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.752730 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.756109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:43.846109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.849099 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.849666 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:43.849836 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.850069 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:43.850528 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:43.851531 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:43.871243 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:43.952368 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.954928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:43.956737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.958865 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:43.959209 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:43.987076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:44.025512 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:44.078538 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.078541 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.081221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.089617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:44.090751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.108554 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:44.214084 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.256346 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:02:44.268986 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.279729 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:44.289223 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.290558 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.292569 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:44.297744 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:44.298454 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.299288 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:44.350453 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:02:44.356949 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:44.410271 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.419113 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.422613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.424202 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.427692 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:44.431037 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:44.448760 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.453984 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:44.463593 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:44.465142 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:44.478407 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:02:44.479036 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:44.482915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:44.483034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.489018 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:44.541052 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:44.600014 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.681268 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.700076 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.732630 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:44.746495 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:44.758523 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.759488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.762960 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.769445 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:44.770192 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.785801 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:44.846851 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.852984 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.854338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.856105 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:44.856643 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.861096 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:44.873158 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:44.928361 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.928828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.930090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:44.940163 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:44.941218 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:44.941476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:44.964838 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:45.020421 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:45.168296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.171662 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.172808 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.174821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.186137 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.190905 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:45.215839 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 02:02:45.234792 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:45.235409 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:02:45.271087 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:45.277323 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:45.278259 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.279358 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.283846 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:45.284482 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.292277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.292783 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:45.311532 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:45.312496 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:45.380088 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.382324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:45.384631 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:02:45.385229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.388088 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:45.395102 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.396840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.497895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:45.500120 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:45.501952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.501984 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.506762 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.509320 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:45.514751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:45.659881 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:45.673405 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:45.693939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.699885 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.703464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.703465 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.709517 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.724246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:45.830305 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.837850 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.838606 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.841683 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:45.842769 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.843017 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.850903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.867290 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.869551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:45.879458 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:45.890769 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:45.893469 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:45.945708 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.092868 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033382
W20240503 02:02:46.095180 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:46.110405 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":20290,"cfile_init":1,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 02:02:46.111228 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7309 bytes on disk
I20240503 02:02:46.111747 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 02:02:46.140988 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:02:46.160178 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.206172 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.208191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:46.209050 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:46.211434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.215287 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.218698 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.219872 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.229158 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:46.245663 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.246146 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.251171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.258742 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:46.266871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:46.297636 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.355809 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.358814 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.358884 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.363065 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:46.364732 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.382757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:46.474553 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:46.482844 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.563896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.565197 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.566519 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.574234 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.575567 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:46.585491 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.600896 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.669415 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.669706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.674715 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.678073 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.688614 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:46.693714 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.694061 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:46.714497 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:46.730377 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:46.778666 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:46.780344 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:46.782164 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.783105 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.791428 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.792683 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:46.843152 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:46.865753 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:46.894585 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.895404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:46.896719 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:46.897647 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:46.903585 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:46.904698 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:46.917654 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:46.922720 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:46.970693 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:47.101377 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:47.104525 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.104704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.105023 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.106056 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:47.116221 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:47.121888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:47.127915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:47.136654 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:47.194232 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:47.214078 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.221081 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.222460 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.230407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.230962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:47.235040 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:47.238662 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:47.269085 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.273586 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:47.273581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.275447 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.293164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.302726 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:47.389153 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:47.515446 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:47.574550 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.575177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.576252 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:47.577337 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.588536 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.591004 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:47.646450 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.653016 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.655615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:47.659547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.667052 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:47.667928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:47.681869 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:47.722371 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.723003 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.727075 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.731938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:47.734777 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:47.735606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:47.745528 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:47.785564 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:02:47.896505 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:47.904978 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:47.951418 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:47.977164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.978201 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:47.984161 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:47.987955 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:47.997217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.005779 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:48.053022 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:48.074797 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.078158 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.082484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.082568 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:48.089708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.100579 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:48.151336 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:48.177326 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.179808 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.190058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.192821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.193795 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:48.203518 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:48.290280 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:48.291555 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.293776 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.297605 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:48.303488 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:48.304478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.304885 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:48.316401 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:48.490171 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.490278 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.495549 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.502935 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:48.506548 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:48.507925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:48.541882 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:48.570940 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:48.578053 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:48.604934 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.615726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.618957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.619917 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:48.625092 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:48.651232 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.651700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.652235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.656395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.666792 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:48.692692 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.699916 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:48.734892 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:48.759138 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:48.790694 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:02:48.977286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.981211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.982472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:48.984535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:48.993242 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:48.993688 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:48.994055 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:49.038184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.039644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.042269 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:49.062691 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.063932 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.068677 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:49.137919 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:49.142402 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.143118 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.144460 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.145320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:49.151911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.160295 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:49.161347 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.162182 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.193396 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.215554 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.248011 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:49.355955 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.358472 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.359025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.362311 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:49.365128 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.370942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:49.429797 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:49.461831 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.465379 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:49.469139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.475028 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.479131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.487454 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:49.520221 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:02:49.520823 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.545948 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 02:02:49.565747 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.567404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.571020 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:49.574674 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.589092 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:49.590907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:49.598030 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.616803 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:02:49.632189 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.663563 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:49.675871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.676492 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:49.676956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.678311 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.688755 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:49.696086 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:49.698315 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:49.710570 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:49.847293 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:49.875945 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.876320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:49.878479 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.878541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.889427 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:49.899487 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:49.948633 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:49.990239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:49.991204 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.005167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.005605 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.007575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.016261 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.056252 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.058379 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.060319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.061815 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.066147 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.072111 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:50.111459 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:50.205363 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:50.245215 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:02:50.256719 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:50.267215 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:50.270902 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:02:50.271605 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:50.313716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.313958 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.324045 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.324667 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.331970 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.336730 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:50.337273 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:50.416292 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.422847 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.423321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.423487 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.430555 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.445605 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.483343 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.483590 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.485563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.487212 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.492398 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.497366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:50.568629 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:50.605850 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:50.609246 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:50.650151 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:50.740471 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.744652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.746392 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.752732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.761370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.778199 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:50.784479 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:50.836971 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.837461 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.838483 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.843974 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.844902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.861058 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.951970 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.952266 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:50.956955 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:50.957644 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:50.959193 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:50.981034 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:51.045543 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:02:51.055686 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:02:51.068846 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.071146 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:51.073241 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:51.074107 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.074306 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:51.080593 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:51.192749 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:51.202893 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:51.207563 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:51.245430 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:51.261363 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.261613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.267503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.269353 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0): perf score=0.033336
I20240503 02:02:51.269651 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:51.276077 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:51.301899 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:51.322319 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.053s user 0.030s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":7685,"dirs.queue_time_us":6614,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":815,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":248,"threads_started":3,"wal-append.queue_time_us":21762}
I20240503 02:02:51.323967 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0): 4583 bytes on disk
I20240503 02:02:51.324791 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.334540 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(b4b734308d3b42958e3a2830a83d44e2): perf score=0.033359
W20240503 02:02:51.356556 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:51.374924 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(b4b734308d3b42958e3a2830a83d44e2) complete. Timing: real 0.040s user 0.026s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":186,"dirs.queue_time_us":6169,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1126,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":221,"threads_started":3,"wal-append.queue_time_us":123}
I20240503 02:02:51.376008 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(b4b734308d3b42958e3a2830a83d44e2): 4838 bytes on disk
I20240503 02:02:51.376924 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(b4b734308d3b42958e3a2830a83d44e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:02:51.455170 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:51.456276 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0): perf score=0.033393
I20240503 02:02:51.458859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:51.459995 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:51.474083 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.483225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:51.494472 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.495260 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:51.498548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:51.520205 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.063s user 0.030s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2404,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":227,"threads_started":3,"wal-append.queue_time_us":9555}
I20240503 02:02:51.521798 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0): 4583 bytes on disk
I20240503 02:02:51.522457 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.523530 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(063a7f2ed28a455697b000f52a4343a3): perf score=0.033410
I20240503 02:02:51.561074 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.564730 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.566112 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(063a7f2ed28a455697b000f52a4343a3) complete. Timing: real 0.042s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":20280,"drs_written":1,"lbm_read_time_us":15244,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":213,"threads_started":2,"wal-append.queue_time_us":333}
I20240503 02:02:51.567037 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3): 4923 bytes on disk
I20240503 02:02:51.567459 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.585667 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:51.601282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:51.611800 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:51.624671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:51.733088 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:51.758448 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:51.782605 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:51.785621 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(d86b733f38fd4f08bd40865b99e2e7b2): perf score=0.033343
W20240503 02:02:51.795087 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:51.803961 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:51.819005 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(d86b733f38fd4f08bd40865b99e2e7b2): perf score=0.033352
I20240503 02:02:51.819293 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(d86b733f38fd4f08bd40865b99e2e7b2) complete. Timing: real 0.033s user 0.031s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":249,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":788,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":580,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":259,"threads_started":3,"wal-append.queue_time_us":150}
I20240503 02:02:51.820411 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2): 4838 bytes on disk
I20240503 02:02:51.824273 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.826898 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(8108933516894d4190ef9a0cf6063be8): perf score=0.033355
I20240503 02:02:51.842002 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(d86b733f38fd4f08bd40865b99e2e7b2) complete. Timing: real 0.023s user 0.014s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":11316,"compiler_manager_pool.run_cpu_time_us":16,"compiler_manager_pool.run_wall_time_us":16,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1689,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":210}
I20240503 02:02:51.841995 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":1694,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":104}
I20240503 02:02:51.842749 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8): 4923 bytes on disk
I20240503 02:02:51.843055 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.843108 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2): 4838 bytes on disk
I20240503 02:02:51.843506 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.878710 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(8108933516894d4190ef9a0cf6063be8): perf score=0.033371
I20240503 02:02:51.897372 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3665,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1667,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":1729}
I20240503 02:02:51.898478 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(e6cc67c73560449e81303a948df8f9df): perf score=0.033374
I20240503 02:02:51.958845 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.060s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1639,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":19449}
I20240503 02:02:51.960058 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8): 4923 bytes on disk
I20240503 02:02:51.960508 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.961625 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df): 4583 bytes on disk
I20240503 02:02:51.962103 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 02:02:51.964622 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.977658 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:51.978986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:51.985219 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:51.988679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:51.990155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.008838 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:52.012681 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:52.013706 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.019419 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:52.028190 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:52.046268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.053516 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.064213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:52.074054 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:52.112222 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:52.142799 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.143321 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.143996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.144601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.144997 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:52.145867 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:52.244611 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:52.322731 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:52.327307 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81): perf score=0.033382
I20240503 02:02:52.343559 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":269,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1608,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":134}
I20240503 02:02:52.344664 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81): 4923 bytes on disk
I20240503 02:02:52.345155 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.345664 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81): perf score=0.033390
I20240503 02:02:52.346005 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(f5767bdfcc6249578f7c3ada3c08da6e): perf score=0.033391
I20240503 02:02:52.370260 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(f5767bdfcc6249578f7c3ada3c08da6e) complete. Timing: real 0.024s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":497,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":3200,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":609,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":398,"threads_started":2,"wal-append.queue_time_us":744}
I20240503 02:02:52.371357 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e): 4583 bytes on disk
I20240503 02:02:52.372033 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.373009 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.027s user 0.016s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":3621,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":994}
I20240503 02:02:52.374142 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(0613ddd56ed346b3b6dfa9e7c3a4335e): perf score=0.033396
I20240503 02:02:52.374748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.376628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.383883 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.384209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.386657 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.392468 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:52.393132 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:52.430410 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(0613ddd56ed346b3b6dfa9e7c3a4335e) complete. Timing: real 0.056s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1199,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3251,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":186}
I20240503 02:02:52.431190 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81): 4923 bytes on disk
I20240503 02:02:52.431566 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.432327 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e): 4838 bytes on disk
I20240503 02:02:52.433322 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 02:02:52.446051 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:52.487520 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.487954 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.493254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.494179 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.502414 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.504463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:52.508859 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:52.561252 15325 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:52.561388 15325 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:02:52.561852 15325 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:02:52.562963 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:52.563146 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:52.563215 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:52.563341 15325 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:52.602686 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.604238 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.607630 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.614199 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:52.617769 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.624475 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:52.651813 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:52.704744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.705355 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.706481 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.708570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.716200 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(8292c06377f445849acdc22d08bd612d): perf score=0.033340
I20240503 02:02:52.716639 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:52.735594 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:52.740794 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(8292c06377f445849acdc22d08bd612d) complete. Timing: real 0.024s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 02:02:52.741587 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2): perf score=0.033346
I20240503 02:02:52.756119 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.014s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1416,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":3066,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":390,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":194}
I20240503 02:02:52.756950 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2): 4838 bytes on disk
I20240503 02:02:52.757357 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.757925 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d): 4583 bytes on disk
I20240503 02:02:52.758289 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240503 02:02:52.788162 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:52.836174 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:02:52.852897 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:52.858713 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:52.874873 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(34e88e69beb64036bb676b771dabd2a4): perf score=0.033382
I20240503 02:02:52.890728 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(34e88e69beb64036bb676b771dabd2a4) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":110,"threads_started":2,"wal-append.queue_time_us":120}
I20240503 02:02:52.891613 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4): 4923 bytes on disk
I20240503 02:02:52.892042 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.919335 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.921408 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.934016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:52.935917 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(34e88e69beb64036bb676b771dabd2a4): perf score=0.033400
I20240503 02:02:52.937785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:52.938078 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:52.945685 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:52.953969 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(34e88e69beb64036bb676b771dabd2a4) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3767,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":299}
I20240503 02:02:52.954967 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2): perf score=0.033404
I20240503 02:02:52.960790 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:52.973085 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1705,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":674}
I20240503 02:02:52.973948 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4): 4923 bytes on disk
I20240503 02:02:52.974324 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:02:52.975034 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2): 4838 bytes on disk
I20240503 02:02:52.975744 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.053968 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.055270 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.061614 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.062251 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:53.063458 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:53.083462 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.083528 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.103194 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.103384 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.108810 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.109505 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.111843 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:53.120369 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:53.140681 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.259999 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(0633596226954c439d2f671c25eebfe6): perf score=0.033341
W20240503 02:02:53.270869 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:53.271724 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(0633596226954c439d2f671c25eebfe6) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1289,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":138}
I20240503 02:02:53.272495 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6): 4838 bytes on disk
I20240503 02:02:53.272835 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240503 02:02:53.316296 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:53.350670 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46): perf score=0.033368
I20240503 02:02:53.369071 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3019,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1341,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":862,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":166}
I20240503 02:02:53.369856 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46): 4921 bytes on disk
I20240503 02:02:53.370208 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.394491 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46): perf score=0.033379
I20240503 02:02:53.409584 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":142}
I20240503 02:02:53.410454 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46): 4921 bytes on disk
I20240503 02:02:53.410800 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.411595 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(0633596226954c439d2f671c25eebfe6): perf score=0.033382
I20240503 02:02:53.427559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.430286 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.434177 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(0633596226954c439d2f671c25eebfe6) complete. Timing: real 0.022s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":6523,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":193}
I20240503 02:02:53.435338 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6): 4838 bytes on disk
I20240503 02:02:53.436372 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.439181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.442411 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:53.443253 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.458508 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.477689 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(490776bd408a48a9bcbbc7a501baa8ce): perf score=0.033401
I20240503 02:02:53.493158 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(490776bd408a48a9bcbbc7a501baa8ce) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1334,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 02:02:53.493849 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce): 4581 bytes on disk
I20240503 02:02:53.494328 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.505069 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.506111 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.507099 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.509076 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:53.518296 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.538295 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.616266 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.617570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.623205 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:53.624145 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.626124 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.643253 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:53.727495 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:53.740417 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:53.749981 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:53.754109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.754202 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:53.756153 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.757423 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.758065 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:53.770582 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:53.779604 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:53.855849 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:53.874878 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d): perf score=0.033352
W20240503 02:02:53.880465 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:53.889218 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:53.890636 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:02:53.892025 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":487}
I20240503 02:02:53.892823 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d): 4838 bytes on disk
I20240503 02:02:53.893224 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 02:02:53.932600 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.932933 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.934513 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:53.946725 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:53.950160 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(613074d3c4514bd5b64a06c59af09a56): perf score=0.033373
I20240503 02:02:53.954546 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:53.965659 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:53.985925 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(613074d3c4514bd5b64a06c59af09a56) complete. Timing: real 0.036s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3506,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1034,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":402}
I20240503 02:02:53.986682 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56): 4584 bytes on disk
I20240503 02:02:53.987010 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240503 02:02:53.993556 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:53.996722 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d): perf score=0.033385
I20240503 02:02:54.008757 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1227,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 02:02:54.009454 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d): 4838 bytes on disk
I20240503 02:02:54.009794 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.024847 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(141097686fe24c8f9956dcc1ad1d253c): perf score=0.033395
I20240503 02:02:54.054839 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(141097686fe24c8f9956dcc1ad1d253c) complete. Timing: real 0.030s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12482,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":138,"threads_started":2,"wal-append.queue_time_us":8379}
I20240503 02:02:54.055658 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c): 4922 bytes on disk
I20240503 02:02:54.056175 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.071244 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.072237 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.074414 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.080161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.096240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.101543 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:54.140615 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:54.156872 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.157617 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.160467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.165700 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:54.177925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.187058 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:54.241030 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:54.270980 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:54.314064 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:54.315990 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.395570 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7): perf score=0.033333
I20240503 02:02:54.404956 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.409272 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.415480 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.417765 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:54.420346 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.422461 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:54.437047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.449021 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.053s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8597,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1431,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":523,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":10075}
I20240503 02:02:54.449936 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7): 4583 bytes on disk
I20240503 02:02:54.450649 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 02:02:54.476194 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.488970 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7): perf score=0.033363
I20240503 02:02:54.511616 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(13a95f88ac1c459ba01be0743297f696): perf score=0.033371
I20240503 02:02:54.513669 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":151}
I20240503 02:02:54.516224 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7): 4583 bytes on disk
I20240503 02:02:54.517890 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.529181 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.530119 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.533464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.539928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.543594 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:54.560029 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.048s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14267,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":666,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":10725}
I20240503 02:02:54.563007 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d): perf score=0.033384
W20240503 02:02:54.568828 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:02:54.569756 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.577345 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.582413 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.019s user 0.012s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":153}
I20240503 02:02:54.583545 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696): 4923 bytes on disk
I20240503 02:02:54.584197 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.585178 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d): 4838 bytes on disk
I20240503 02:02:54.585567 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.594990 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.600306 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.603982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.607282 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:54.613525 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:54.613761 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.616420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:54.642539 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:02:54.658423 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:02:54.670796 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.827375 21713 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:02:54.827518 21713 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:02:54.827983 21713 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:02:54.828891 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:54.829190 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:54.829259 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:02:54.829397 21713 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:02:54.839231 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:54.861828 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:54.874341 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.876976 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.879995 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.894224 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.904516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.921073 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:54.947880 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.954082 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(79c5f35eb8b546de8015c5235913efce): perf score=0.033336
I20240503 02:02:54.962620 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:54.966632 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:54.985023 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:54.986275 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:54.986634 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:54.988108 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.033s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1570,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":123}
I20240503 02:02:54.988922 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce): 4923 bytes on disk
I20240503 02:02:54.989297 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 02:02:54.989960 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(ae1f9c8efd2040d6842c6b9f7c6173aa): perf score=0.033344
I20240503 02:02:54.991211 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:55.005609 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(ae1f9c8efd2040d6842c6b9f7c6173aa) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":142}
I20240503 02:02:55.006433 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa): 4583 bytes on disk
I20240503 02:02:55.006880 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 02:02:55.008404 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:55.073827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.077989 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.078111 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.079576 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.087705 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(79c5f35eb8b546de8015c5235913efce): perf score=0.033373
I20240503 02:02:55.098091 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:55.113471 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1317,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1792,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":116}
I20240503 02:02:55.114306 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce): 4923 bytes on disk
I20240503 02:02:55.114645 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 02:02:55.116472 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:55.168442 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:55.171347 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:55.235714 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.238615 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.239077 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.239451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:55.242002 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:55.242125 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:55.300897 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:55.369578 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:02:55.370261 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:55.388120 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:02:55.389062 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:55.400926 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.405856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.407197 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.408303 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.411325 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:55.419546 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:55.427613 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:55.433661 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:55.500181 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:55.508816 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(0f3bdd27c750447a87691c0f57a70ee1): perf score=0.033349
I20240503 02:02:55.522009 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88): perf score=0.033350
I20240503 02:02:55.522109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.522337 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.527057 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.541085 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.541395 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.019s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3853,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":274}
I20240503 02:02:55.542317 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88): 4838 bytes on disk
I20240503 02:02:55.542793 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 02:02:55.552542 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(0f3bdd27c750447a87691c0f57a70ee1) complete. Timing: real 0.043s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4524,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1115,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":151}
I20240503 02:02:55.553452 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(0f3bdd27c750447a87691c0f57a70ee1): 4923 bytes on disk
I20240503 02:02:55.553825 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(0f3bdd27c750447a87691c0f57a70ee1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 02:02:55.557487 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:55.559963 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling FlushMRSOp(04582c3f86d744ccbc96eec50a10994f): perf score=0.033360
I20240503 02:02:55.567368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:55.578219 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:55.582854 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: FlushMRSOp(04582c3f86d744ccbc96eec50a10994f) complete. Timing: real 0.023s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4972,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2253,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":461,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":2731}
I20240503 02:02:55.583724 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f): 4583 bytes on disk
I20240503 02:02:55.584197 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:02:55.616811 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling FlushMRSOp(04582c3f86d744ccbc96eec50a10994f): perf score=0.033375
I20240503 02:02:55.622974 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.623212 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.624301 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:55.624931 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.630647 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.636222 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: FlushMRSOp(04582c3f86d744ccbc96eec50a10994f) complete. Timing: real 0.019s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":118}
I20240503 02:02:55.638442 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f): 4583 bytes on disk
I20240503 02:02:55.638831 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 02:02:55.652453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:55.864044 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:55.865978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.867626 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.868314 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.870927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:55.871366 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:55.884073 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:55.937708 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:55.989794 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.995376 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.996198 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:55.996577 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.005779 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.012784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.015908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.028183 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:56.035876 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.036046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.044724 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(a105660c12094c71b3a4b80e661d9308): perf score=0.033384
I20240503 02:02:56.054631 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:56.055624 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(662782d61c6f472baab4d613b7e6f5cd): perf score=0.033388
I20240503 02:02:56.062450 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.085283 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(a105660c12094c71b3a4b80e661d9308) complete. Timing: real 0.040s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4225,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":164,"threads_started":2,"wal-append.queue_time_us":11827}
I20240503 02:02:56.086139 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308): 4923 bytes on disk
I20240503 02:02:56.086485 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:02:56.087137 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(1c464b6e30bb414bbe70ed030c5f481c): perf score=0.033394
I20240503 02:02:56.089848 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(662782d61c6f472baab4d613b7e6f5cd) complete. Timing: real 0.034s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8060,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":5432}
I20240503 02:02:56.090793 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(662782d61c6f472baab4d613b7e6f5cd): 4838 bytes on disk
I20240503 02:02:56.091164 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(662782d61c6f472baab4d613b7e6f5cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 02:02:56.102959 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(1c464b6e30bb414bbe70ed030c5f481c) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1117,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":168}
I20240503 02:02:56.103991 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(1c464b6e30bb414bbe70ed030c5f481c): 4583 bytes on disk
I20240503 02:02:56.105581 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(1c464b6e30bb414bbe70ed030c5f481c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 02:02:56.106071 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:02:56.162315 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:56.285200 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:56.329892 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:56.330771 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:56.390622 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.399466 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.401168 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.402525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.407590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.415184 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.417301 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:56.427366 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.438810 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.439075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:56.441238 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.445351 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.447342 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:56.478019 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:56.520259 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:02:56.542503 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.543828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.546463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.547503 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.548301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.562669 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:56.708711 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.752041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.754990 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:56.756161 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.759322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.761724 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.764286 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:56.770079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:56.795966 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:56.799259 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.863855 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.866703 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.866824 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.867115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.869103 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.879981 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:56.914268 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.968469 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.971328 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:56.971371 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:56.972167 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:56.976089 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:56.989799 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:56.991281 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:57.027262 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:57.028095 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:57.074733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.076150 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.080173 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.083513 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.085520 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:57.100003 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:57.101549 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:57.133026 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:57.275686 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:57.279170 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:57.281524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.283109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:57.284881 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.285658 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.288525 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.289672 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:57.369318 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:57.396311 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.398543 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.401459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.410326 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:57.417605 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:57.424984 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:57.434526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.436570 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:57.445969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.447273 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.458878 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.466137 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:57.482825 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:57.531538 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:57.580250 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:02:57.680969 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:57.703258 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:57.788110 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.788228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.801532 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.801770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.810952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:57.812391 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:57.816390 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:57.833544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:57.835245 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.855986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.859717 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.864171 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:57.866827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:57.886096 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:57.958274 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:57.963677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.965173 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.966488 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:57.968088 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:57.968680 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:57.995703 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:58.004879 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:58.076553 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:58.159601 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.160552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.161551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.164465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.168977 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:58.180312 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:58.234851 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:58.244927 22685 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:02:58.245049 22685 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:02:58.245532 22685 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:02:58.246273 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:02:58.246773 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:58.246845 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:02:58.247197 22685 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:02:58.278012 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.278276 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.278579 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.282711 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:58.285171 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.291452 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:58.358227 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:58.380986 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.384624 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.388473 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.389237 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.390492 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:58.403780 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:58.480558 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:58.491434 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:58.496085 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.497380 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.498730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.503064 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.521822 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:58.689473 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.695294 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:58.696123 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:58.696786 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.703909 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:58.704809 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.711534 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:58.722904 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:58.727874 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:58.746730 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:58.807482 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.810109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.811182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.817535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.826857 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:58.840643 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:58.857036 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:58.877185 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.879211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:58.882372 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.884281 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:58.891078 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:58.892617 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:59.003147 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.142693 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:59.143087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.143409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.143448 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.155121 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:59.167490 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:59.204190 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.240545 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.244017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:59.245631 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:59.248557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.258320 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.262953 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.267549 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:59.288501 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:59.297890 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.305747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.307581 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:02:59.313495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:59.315150 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:59.315475 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:02:59.316422 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.316695 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.325752 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:02:59.363118 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:59.433333 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:02:59.463413 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:02:59.597257 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.597748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.602042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:59.602941 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:59.616086 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.618157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:02:59.651106 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:02:59.655745 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.656456 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.666023 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:59.668614 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:59.669503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:02:59.670805 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 02:02:59.678805 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.688266 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.690488 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.752493 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:02:59.772228 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.772977 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 02:02:59.773248 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:02:59.778285 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:02:59.784917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:02:59.786954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:02:59.787814 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:02:59.793102 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:02:59.806092 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:02:59.900861 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(063a7f2ed28a455697b000f52a4343a3): perf score=0.033387
I20240503 02:02:59.932914 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(063a7f2ed28a455697b000f52a4343a3) complete. Timing: real 0.032s user 0.028s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":167,"dirs.queue_time_us":4355,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":261,"threads_started":3,"wal-append.queue_time_us":172}
I20240503 02:02:59.933992 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(79c5f35eb8b546de8015c5235913efce): perf score=0.033375
I20240503 02:02:59.954092 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.020s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1141,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":500,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":191}
I20240503 02:02:59.955065 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3): 4923 bytes on disk
I20240503 02:02:59.955700 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 02:02:59.956478 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce): 4923 bytes on disk
I20240503 02:02:59.956842 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:02:59.970780 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(e6cc67c73560449e81303a948df8f9df): perf score=0.033377
W20240503 02:02:59.985687 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:02:59.989930 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:03:00.001605 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.012915 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.022413 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.028925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.029103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:00.030623 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:00.059921 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.089s user 0.029s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":695,"dirs.queue_time_us":57466,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":3938,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":227,"threads_started":3,"wal-append.queue_time_us":36447}
I20240503 02:03:00.061285 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(79c5f35eb8b546de8015c5235913efce): perf score=0.033401
I20240503 02:03:00.092769 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.031s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1160,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":1,"wal-append.queue_time_us":199}
I20240503 02:03:00.093684 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(0613ddd56ed346b3b6dfa9e7c3a4335e): perf score=0.033384
I20240503 02:03:00.121474 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(0613ddd56ed346b3b6dfa9e7c3a4335e) complete. Timing: real 0.028s user 0.023s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1054,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":2249}
I20240503 02:03:00.122689 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce): 4923 bytes on disk
I20240503 02:03:00.123432 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(79c5f35eb8b546de8015c5235913efce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.124462 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e): 4838 bytes on disk
I20240503 02:03:00.125478 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.127051 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df): 4583 bytes on disk
I20240503 02:03:00.127399 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.128360 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d): perf score=0.033334
I20240503 02:03:00.153424 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.025s user 0.022s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":125,"threads_started":1,"wal-append.queue_time_us":1368}
I20240503 02:03:00.154505 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d): 4838 bytes on disk
I20240503 02:03:00.154915 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(b39fa1cb3863414287a47a0832c8b55d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:03:00.186313 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:00.186900 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.191628 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.197957 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:00.201400 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:00.208329 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88): perf score=0.033392
W20240503 02:03:00.229260 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:03:00.233039 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.233850 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.260380 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.052s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":34050,"compiler_manager_pool.run_cpu_time_us":14,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1205,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":4064}
I20240503 02:03:00.261330 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(141097686fe24c8f9956dcc1ad1d253c): perf score=0.033347
I20240503 02:03:00.277449 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(141097686fe24c8f9956dcc1ad1d253c) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1168,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":181}
I20240503 02:03:00.278604 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c): 4922 bytes on disk
I20240503 02:03:00.280230 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.281075 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88): 4838 bytes on disk
I20240503 02:03:00.281369 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.347826 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.348495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.355034 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.358877 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:00.359136 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:00.359474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:00.365089 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:00.384938 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:00.403708 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:03:00.411613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.412334 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.418040 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.421995 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:00.424626 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:03:00.432278 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:00.439539 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:00.449359 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:00.483512 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:00.523090 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:00.532555 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d): perf score=0.033362
I20240503 02:03:00.544356 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1160,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":138}
I20240503 02:03:00.545110 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d): 4838 bytes on disk
I20240503 02:03:00.545483 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 02:03:00.652987 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:03:00.657265 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(490776bd408a48a9bcbbc7a501baa8ce): perf score=0.033387
I20240503 02:03:00.678840 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(490776bd408a48a9bcbbc7a501baa8ce) complete. Timing: real 0.021s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4311,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":314}
I20240503 02:03:00.679711 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(613074d3c4514bd5b64a06c59af09a56): perf score=0.033385
I20240503 02:03:00.698165 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.699836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.702455 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:00.707455 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.709034 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.715183 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(613074d3c4514bd5b64a06c59af09a56) complete. Timing: real 0.035s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1774,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1653,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":523,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":8257}
I20240503 02:03:00.716152 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56): 4584 bytes on disk
I20240503 02:03:00.716562 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.717222 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(13a95f88ac1c459ba01be0743297f696): perf score=0.033379
I20240503 02:03:00.733876 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":211}
I20240503 02:03:00.734561 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696): 4923 bytes on disk
I20240503 02:03:00.734933 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.735580 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:00.735625 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce): 4581 bytes on disk
I20240503 02:03:00.740087 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.796969 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(8108933516894d4190ef9a0cf6063be8): perf score=0.033379
I20240503 02:03:00.807298 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.807394 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.815981 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:00.816236 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.824642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:00.825480 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:00.836480 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.039s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1207,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":657,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":112,"threads_started":1,"wal-append.queue_time_us":207}
I20240503 02:03:00.837527 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(f5767bdfcc6249578f7c3ada3c08da6e): perf score=0.033370
I20240503 02:03:00.841290 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:00.855010 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(f5767bdfcc6249578f7c3ada3c08da6e) complete. Timing: real 0.017s user 0.012s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1360,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":160}
I20240503 02:03:00.856066 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8): 4923 bytes on disk
I20240503 02:03:00.856436 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(8108933516894d4190ef9a0cf6063be8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.857218 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e): 4583 bytes on disk
I20240503 02:03:00.857565 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:03:00.887256 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.888370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:00.889771 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.895911 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:00.900668 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:00.914192 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:00.925585 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:03:00.972236 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:01.098703 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.108834 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7): perf score=0.033366
I20240503 02:03:01.126333 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":377,"threads_started":1,"wal-append.queue_time_us":469}
I20240503 02:03:01.127103 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7): 4583 bytes on disk
I20240503 02:03:01.127418 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(6ad545d33bbc40458c75ed7c90d92ba7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 02:03:01.139047 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.141292 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.143363 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.147087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.149237 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:01.160068 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:01.173064 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.234393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.235373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.237635 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.244025 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.246510 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.258883 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:01.337836 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.344743 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.345170 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.346218 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.346951 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.353545 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.368685 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:01.378835 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81): perf score=0.033383
I20240503 02:03:01.406045 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.027s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":205,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1218,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":142}
I20240503 02:03:01.406785 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81): 4923 bytes on disk
I20240503 02:03:01.407204 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(c361f82c69ef477d9c3cfd65e07d7f81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 02:03:01.423660 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:01.454816 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.458833 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.459887 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:01.461863 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.463021 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:01.476863 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:01.477578 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.495223 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(8292c06377f445849acdc22d08bd612d): perf score=0.033376
I20240503 02:03:01.509523 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(8292c06377f445849acdc22d08bd612d) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1292,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":120,"threads_started":2,"wal-append.queue_time_us":109}
I20240503 02:03:01.510223 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d): 4583 bytes on disk
I20240503 02:03:01.510591 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240503 02:03:01.588122 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.657811 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.659404 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.664947 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.668485 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.668798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:01.669270 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.693892 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:01.756366 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.789183 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.793262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.795462 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.804805 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.807195 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.809417 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:01.813367 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:01.815910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:01.824899 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.833544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:01.834872 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:01.841517 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.848613 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:01.850824 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:01.868211 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:01.909514 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46): perf score=0.033370
I20240503 02:03:01.921391 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1218,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":132}
I20240503 02:03:01.922116 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46): 4921 bytes on disk
I20240503 02:03:01.922493 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(3b829f779c8a459597c47dc3e42a4c46) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 02:03:01.930100 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:01.948632 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:03:01.983673 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:02.012795 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0): perf score=0.033386
W20240503 02:03:02.016537 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:02.025386 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":105}
I20240503 02:03:02.026072 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0): 4583 bytes on disk
I20240503 02:03:02.026391 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(da7d3c59eeb949ec84db013f24e656b0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 02:03:02.041924 22685 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:02.042035 22685 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:02.042395 22685 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:02.042989 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:02.043196 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:02.043270 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:02.043386 22685 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:02.065912 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:02.154525 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:02.165364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.171018 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.171473 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.174307 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2): perf score=0.033375
I20240503 02:03:02.175472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.193706 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.194289 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:02.206702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.208969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:02.209151 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.035s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":529}
I20240503 02:03:02.217898 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2): 4838 bytes on disk
W20240503 02:03:02.218652 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:02.218657 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(097947dce87b409ca4a57cb752fdd5e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 02:03:02.219559 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.224789 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.239919 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.242727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.331902 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.333458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.339360 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.341779 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.346241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.352838 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:02.360072 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:02.413821 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:02.489353 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:02.510668 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.511909 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.512635 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.514320 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:02.514492 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.536679 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:02.602838 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:02.605471 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:03:02.646289 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.648672 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.652009 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:02.657884 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.664814 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.671625 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:02.676405 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:02.750743 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.753769 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.762655 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.763219 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:02.771481 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.780972 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:02.793835 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:02.796789 23546 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:02.796897 23546 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:02.797281 23546 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:02.798170 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:02.798249 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:02.798326 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:02.798453 23546 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:02.862010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.862557 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.863117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:02.869359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:02.870826 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:02.871801 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:02.989828 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:03.040977 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:03.079489 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.080037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.082263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.088057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.088974 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:03.092496 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:03.128671 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:03.158548 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:03.189646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.192128 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.196249 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.198503 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:03.203320 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:03.207091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.207571 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.238240 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.243407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.243880 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.250288 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.255342 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.258430 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:03.446864 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:03.501078 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:03.537948 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:03.542475 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.546099 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.546489 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.546921 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:03.561627 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.568230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.608378 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.610049 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.610134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.611259 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.620702 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.624330 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:03.638613 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:03.702955 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:03.710793 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.711069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.713991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.714999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.722098 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.738967 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:03.740060 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:03.855351 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:03.856940 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.858618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.867240 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:03.876886 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:03.881829 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:04.023538 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.024287 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.027314 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.035842 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:04.047294 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:04.049161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:04.049571 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:04.056272 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:04.109587 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.113040 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:04.137799 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.141037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.142544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.151996 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.160385 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.168188 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:04.182691 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.229207 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:04.235138 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.238317 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.242700 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.248333 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.256706 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:04.259534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:04.277755 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.379170 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.392110 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:03:04.420244 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:04.462841 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.464758 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.465860 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.466558 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.469077 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.473752 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:04.514719 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.563318 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.566851 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:04.574431 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.574960 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.595988 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.599210 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:04.608608 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.609241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.610885 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.614121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.617465 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:04.620570 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:04.629789 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.629957 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:04.632104 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:04.670526 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:04.672327 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:03:04.681561 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:03:04.724968 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:04.775722 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:03:04.810228 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:04.828148 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:04.956087 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.958443 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.958529 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:04.962002 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:04.967375 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:04.981519 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:04.993506 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:05.005646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.009383 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.009663 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.023449 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.034691 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.116812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.119725 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.120465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.121218 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.123598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.131533 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:05.149811 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:05.308188 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.310405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.311621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.312502 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.314070 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:05.320205 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:05.327453 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:05.387851 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:05.398707 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:05.426064 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.426652 22685 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:05.426766 22685 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:05.427171 22685 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:05.428406 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:05.428498 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:05.428575 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:05.428865 22685 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:05.430068 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:05.432572 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.432695 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.433192 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.433277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:05.457617 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:03:05.507498 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:05.531571 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.534799 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.536988 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.542433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.548370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.552830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:05.568029 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:05.606608 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:03:05.607259 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:05.632751 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.636756 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.638849 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.641670 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:05.645328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.648288 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:05.750985 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:05.761197 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:03:05.771059 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:05.810252 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:05.851833 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.852931 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.853396 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:05.853972 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.855713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.864090 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:05.963200 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:05.971277 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.971647 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:05.978076 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:05.980110 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:05.988370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:06.006182 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.021858 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:06.022291 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.028924 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.029322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.034003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:06.042165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:06.119221 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:06.232896 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:06.313750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.315155 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.321208 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.321614 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:06.340728 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:06.341470 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:06.346258 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:06.384371 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.384845 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.388816 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:06.405033 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:06.411721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.414970 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:06.415351 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:06.499244 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:06.499444 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:06.500059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.500907 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.504418 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:06.508908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.518559 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:06.615186 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:06.617364 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:06.653069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.653735 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:06.656683 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.663038 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:06.663656 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:06.666158 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:06.683925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:06.693269 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:06.723572 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:06.805761 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.809084 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.815418 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:06.816874 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:06.818202 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:06.818488 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:06.856348 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:06.911636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:06.912376 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.914216 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.918424 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:06.923346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:06.952677 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:06.985093 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:06.987002 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:07.015998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.017308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.017800 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.018994 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:07.023473 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:03:07.028852 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:07.032194 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:07.057427 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:07.098143 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:07.124264 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:07.215348 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:07.235556 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.238636 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.245488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.246433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.258615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:07.258643 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:07.326602 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:07.344647 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.346206 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.350023 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.357782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.373203 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:07.392266 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.400308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.407444 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.411312 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:07.415488 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:07.421157 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:07.427944 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.442767 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:07.549793 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:07.603060 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:07.707438 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.713220 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.713869 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.715521 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:07.722352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.734597 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:07.744061 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:07.777699 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.779959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.784824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.786290 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:07.787196 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:07.802667 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:07.866286 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:07.884289 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.884992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:07.885175 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:07.889271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:07.890492 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:07.901705 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.032302 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.033808 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.034451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.036538 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.045713 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.064555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:08.069202 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.127149 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.130270 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:08.174367 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:08.190431 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.193902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.197829 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.201467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:08.213196 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:08.217793 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.218128 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:08.274376 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:08.300736 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.301862 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.304375 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.305008 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.306769 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.312665 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:08.329427 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:08.383939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.387362 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.390545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.399088 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.401007 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.409073 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:08.486588 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.531200 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:03:08.541692 26171 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:08.541863 26171 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:08.542276 26171 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:08.543265 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:08.543380 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:08.543458 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:08.543591 26171 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:08.599674 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:08.619096 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.619164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.623232 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.623956 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.631356 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.637166 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.719297 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.721359 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.726310 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.739924 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.740824 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.763341 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.771183 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.787879 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:08.787940 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:08.794554 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:08.796896 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:08.810171 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:08.831439 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.905377 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:08.923573 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.934136 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:08.971585 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:09.045354 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:09.068600 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:09.096879 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.101524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.103464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.104602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.129456 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.133960 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:09.144665 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.150876 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.167266 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.169518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.174801 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:09.175015 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:09.190210 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:09.238961 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(86e314466c01488bb661c596c405db51): perf score=0.033340
W20240503 02:03:09.252585 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:09.257241 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(86e314466c01488bb661c596c405db51) complete. Timing: real 0.018s user 0.014s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3615,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1379,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":260}
I20240503 02:03:09.258186 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(86e314466c01488bb661c596c405db51): 4838 bytes on disk
I20240503 02:03:09.258559 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(86e314466c01488bb661c596c405db51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240503 02:03:09.269061 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.271559 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.271814 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.274783 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.290403 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.297021 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:09.308069 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:09.347282 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:09.416842 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:09.440580 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.444386 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.445959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:09.447551 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.457530 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.457671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:09.466897 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:09.562677 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:03:09.569473 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:09.582434 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.586479 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.588125 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.590072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.595917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.607715 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:09.651736 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:09.686787 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.692025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.692647 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:09.694159 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.698062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:09.706701 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:03:09.707410 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:09.713352 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:09.761355 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:03:09.785151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.785363 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.788452 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:09.791280 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:09.796169 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:09.796646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:09.903204 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:03:09.903877 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:10.007499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.008431 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.012017 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.013223 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.018901 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.032229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.037539 25234 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:10.037654 25234 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:10.038035 25234 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:03:10.044916 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:10.044988 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:10.045325 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:10.045411 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:10.045578 25234 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:10.124537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.125190 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.128057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.130136 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.133343 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.150795 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.168078 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.168087 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.179646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.195081 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.200781 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.201752 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:10.205309 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:10.212431 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:10.240933 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:10.393137 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:10.490461 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.490974 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.498986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.499374 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.507139 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.514076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:10.532708 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:10.533313 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:10.548843 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.558200 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88): perf score=0.033386
I20240503 02:03:10.562615 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.569026 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.572777 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.573423 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.582155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.588757 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.030s user 0.015s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2522,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1222,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1043,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":123}
I20240503 02:03:10.589635 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88): 4838 bytes on disk
I20240503 02:03:10.590206 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(fdbd697a90534ff3b7a5e17780207d88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 02:03:10.601039 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:03:10.662150 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:10.670010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.673084 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.674556 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.676556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.677282 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.686969 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:10.687060 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:10.703192 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 02:03:10.706079 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:10.798754 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:10.813041 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:10.826490 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.828986 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:10.830659 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:10.831686 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.832463 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.839890 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:10.856626 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:10.961995 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:03:10.987254 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:10.988118 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:10.990936 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.001395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.002692 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.008378 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.098383 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.099272 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.107401 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.110469 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.116611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:11.126302 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.185773 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:11.187440 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.189091 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.198431 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.198590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.202996 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:11.242883 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:11.264876 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling FlushMRSOp(13a95f88ac1c459ba01be0743297f696): perf score=0.033372
I20240503 02:03:11.276746 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: FlushMRSOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":401,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":115}
I20240503 02:03:11.277491 14849 maintenance_manager.cc:419] P 754fc04ea9914073a56c0cf164f63240: Scheduling UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696): 4923 bytes on disk
I20240503 02:03:11.277804 14739 maintenance_manager.cc:643] P 754fc04ea9914073a56c0cf164f63240: UndoDeltaBlockGCOp(13a95f88ac1c459ba01be0743297f696) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 02:03:11.360736 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.409792 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:11.419330 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.420382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.427830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.431013 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.435427 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.437424 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:11.463163 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.468031 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.493129 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.509603 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:11.528324 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.531721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:11.532728 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.534235 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.544765 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.563170 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.582123 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.583532 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.593415 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:11.594213 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:11.594579 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.599231 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:11.680747 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.697527 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.735842 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:11.752857 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:11.836989 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:03:11.852993 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:11.862504 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.864950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.867432 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.872650 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:11.873580 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.878517 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:11.879084 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:11.882824 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling FlushMRSOp(e6cc67c73560449e81303a948df8f9df): perf score=0.033400
I20240503 02:03:11.901360 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: FlushMRSOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":162,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":1476}
I20240503 02:03:11.902281 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df): 4583 bytes on disk
I20240503 02:03:11.902674 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(e6cc67c73560449e81303a948df8f9df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:03:11.971503 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.983832 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:11.985847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.995050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:11.998128 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.003002 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:12.031343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.031527 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.034775 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.035872 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:12.038285 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.042794 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:12.181075 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:12.265870 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:12.289105 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:12.365064 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.367537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.367538 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:12.377094 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:12.380263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.381624 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.390216 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:12.412560 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:12.416718 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.431087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.431538 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.432209 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:12.433758 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.436607 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:12.443975 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:03:12.521183 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:12.533051 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.534561 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.537026 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:12.537187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.547655 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.560957 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:12.661680 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling FlushMRSOp(ae1f9c8efd2040d6842c6b9f7c6173aa): perf score=0.033336
W20240503 02:03:12.667814 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:12.681032 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: FlushMRSOp(ae1f9c8efd2040d6842c6b9f7c6173aa) complete. Timing: real 0.019s user 0.014s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":2503}
I20240503 02:03:12.682068 13085 maintenance_manager.cc:419] P 22b582a5f4774861a5619c51198d39c5: Scheduling UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa): 4583 bytes on disk
I20240503 02:03:12.682482 12975 maintenance_manager.cc:643] P 22b582a5f4774861a5619c51198d39c5: UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 02:03:12.694972 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.697638 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.701141 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.708362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.709226 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:12.709748 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:12.776139 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:12.847800 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d): perf score=0.033392
I20240503 02:03:12.854060 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.854792 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.855564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:12.860359 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:12.867599 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.020s user 0.013s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":3970,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":121}
I20240503 02:03:12.872395 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d): 4838 bytes on disk
I20240503 02:03:12.872431 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.872939 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(007feed148fb44e9a6fe9d449b6d8f9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:03:12.876533 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling FlushMRSOp(a105660c12094c71b3a4b80e661d9308): perf score=0.033346
I20240503 02:03:12.890200 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:12.895466 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:12.905673 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: FlushMRSOp(a105660c12094c71b3a4b80e661d9308) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":434,"dirs.run_wall_time_us":1693,"drs_written":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4,"lbm_write_time_us":509,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":161}
I20240503 02:03:12.906579 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308): 4923 bytes on disk
I20240503 02:03:12.906996 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 02:03:12.974853 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.975452 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:12.980444 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:12.980834 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:12.989459 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:12.991236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.007401 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:13.009292 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.021029 25234 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:13.021145 25234 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:13.021550 25234 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:03:13.023423 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:13.023829 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:13.023900 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:13.024040 25234 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:13.058786 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.059736 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.061414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.065528 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.068260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:13.088316 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:13.089402 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:13.125211 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.295563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.299717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.305418 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.305820 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.306450 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:13.317917 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:13.318694 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:13.353513 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:13.362711 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:13.407976 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.409332 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.410604 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.436489 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.439671 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.443033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.443495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.448439 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:13.448823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:13.457047 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.462010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:13.468793 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.490712 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.491075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:13.769657 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:13.774869 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.781544 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:13.786778 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.788923 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.790411 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.791342 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.825716 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.845124 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.845788 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.848390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:13.861619 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:13.865958 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.878413 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:13.887332 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:13.891008 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:13.972287 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.973743 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:13.975247 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.976060 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:13.979593 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:13.980034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:14.008246 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.051877 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.103238 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:14.127547 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.129889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.131551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.132622 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.140468 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.144026 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:14.153612 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:14.257531 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.275013 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:14.286325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.291412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:14.295322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.300002 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.300280 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.315382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:14.390259 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:03:14.399124 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.401551 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.411053 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.412485 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.415284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.440143 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:14.486392 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.486766 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.490850 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.493310 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:14.493989 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.495193 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:14.508579 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:14.510066 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.525532 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.600081 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.730401 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:14.732354 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.733860 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.734462 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.745973 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.754961 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:14.761241 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.856508 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.860260 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.865175 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:14.873975 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:14.874770 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:14.875644 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:14.877713 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:03:14.878515 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:14.892979 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.901113 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:14.902099 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:14.907155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.907465 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.908742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:14.909094 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:14.913065 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:03:14.916541 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:14.928804 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:03:14.970489 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:15.001222 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:15.040234 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:15.234549 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.241493 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.247098 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:15.249799 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.254189 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:15.255972 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:15.271027 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:15.296597 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.298462 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.301234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:15.303563 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.306447 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.308250 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:15.398649 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:15.401623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.402997 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.406405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.407786 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.420665 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:15.431339 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:15.462890 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:15.504555 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:15.566924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.566998 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.570621 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.584471 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:15.584867 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:15.587296 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:15.659250 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:15.708093 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:15.716950 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.717445 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.718012 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.720888 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.727783 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:15.728888 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:15.743918 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:03:15.775296 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:03:15.782333 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:15.819384 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.821300 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.825935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.826747 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:15.826838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.838914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:15.857213 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:03:15.869004 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:15.917268 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.917268 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.923890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:15.924573 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:15.932339 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:15.934691 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:15.967235 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:16.008059 27653 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:16.008188 27653 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:16.008616 27653 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:16.009264 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:16.009521 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:16.009589 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:16.009752 27653 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:16.011387 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:16.126976 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.129722 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.131291 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.132283 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.138842 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.146476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.245465 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.255306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.256422 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.260069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.260721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.262280 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.262457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.269037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.272207 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.280345 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.307881 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.320720 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:16.330271 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:16.394897 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.402220 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.411919 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:16.457458 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.460850 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.578732 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:16.615671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.619513 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.621788 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.632125 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.634832 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.640542 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.640830 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.651566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.661360 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.672958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.676801 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.685750 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.776350 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.777237 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.777714 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.784879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.791119 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.803274 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:16.803944 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.853826 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:16.856946 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:16.869415 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:16.878755 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:03:16.981335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:16.981519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.983815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:16.983911 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:16.985567 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:16.986045 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:17.046929 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:17.086726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.089417 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.090258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.098034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:17.101557 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:17.102286 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:17.111716 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:17.192504 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.193393 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.197146 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.207933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.208319 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:17.212463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:17.232156 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:17.233042 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:17.291843 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:17.299257 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.299446 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.303685 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.304715 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.308125 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:17.327958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:17.328374 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:17.400255 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:17.479669 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:17.508373 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.512100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.515980 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:17.516075 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:17.516510 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.517551 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.627576 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.633968 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.639655 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.646135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:17.646751 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:17.646873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.649422 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:17.650482 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:17.661193 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.662251 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:17.669205 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:17.690837 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:17.692086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:17.727284 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:17.793808 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:17.818339 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:17.885730 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:17.901553 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:17.995741 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.004842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.010326 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.013459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.013988 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.029100 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.034082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:18.037895 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:18.040274 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.057801 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.064553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.067354 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.087692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:18.098285 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:18.165416 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.165796 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.169674 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.171962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.173274 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.198087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.274113 25234 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:18.274230 25234 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:18.274602 25234 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:03:18.275552 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:18.275894 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:18.275981 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:18.276121 25234 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:18.281491 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:03:18.360136 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.361997 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.362463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.364730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.372542 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.378454 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:18.423156 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:18.428824 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:18.468966 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:18.482163 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.486284 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.486490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.490243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.490496 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.491966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:18.506206 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:18.592778 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.594779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.596374 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.600055 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.609320 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.610051 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.695031 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:18.695457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.695626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.697530 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.702314 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.703433 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:18.805706 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:18.890039 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:18.901364 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.902318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:18.906071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:18.907194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:18.908327 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:18.914014 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:18.915539 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:18.928259 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:18.934634 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.014997 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.017586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.022728 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.026242 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.029407 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.036440 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.078754 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.081887 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.086465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.088248 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.088719 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.091957 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.102167 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.169104 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:19.263345 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.349505 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.353356 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.354621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.359890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.360828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.369336 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:19.387549 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.388762 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:19.399807 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:19.441252 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.442499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.446918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.447430 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.447826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.458110 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.475803 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.543339 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.546229 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.551554 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.552374 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.553968 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.563787 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.569823 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:03:19.654434 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:19.659374 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.675439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.682024 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.682430 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.682629 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.683411 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.694056 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.752130 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:19.851029 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.856370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.856750 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.859798 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:19.860055 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.866381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:19.875350 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:19.895056 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:19.957274 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:03:19.964768 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.967262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:19.970605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:19.972056 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:03:19.979102 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:19.981071 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:19.981093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:19.992331 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.042279 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.049319 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.049722 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.062676 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:20.064242 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.082526 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:20.100277 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:20.109100 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:20.155674 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:03:20.200214 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:20.207403 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:20.262266 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:20.279109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.281018 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:20.284066 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:20.288393 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:20.289510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.291291 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.299448 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:20.381866 29385 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:20.381999 29385 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:20.382439 29385 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:20.385353 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:20.385576 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:20.385656 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:20.385859 29385 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:20.396674 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.401939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.412127 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:20.412526 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:20.417155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.425069 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.425711 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:20.425971 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.429178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:20.437119 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.448529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.453694 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:20.464969 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:20.472451 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:20.530858 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:20.598781 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:20.761932 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.762480 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.767746 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.768034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.790486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:20.794569 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:20.797819 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:20.838922 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:20.839071 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:03:20.841897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.841922 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:20.842160 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.842355 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.845152 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.867871 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:20.942081 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.949884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:20.953531 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.960645 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:20.960680 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:20.968897 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:20.972648 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 02:03:21.034494 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:21.091137 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:03:21.095432 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:21.095652 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:21.099241 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.100409 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.107646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.109014 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.124130 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:21.239334 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:21.261174 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.262056 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.262362 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.264897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.265422 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.281167 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:21.362131 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:21.372982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.373057 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.375247 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:21.377588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.390224 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.394277 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.458900 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.464087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:21.464726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.470580 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.472177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.478395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:21.503738 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:21.518846 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:21.529186 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:21.670153 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:21.672906 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:21.676789 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:21.694810 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.695621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.696116 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.700464 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.708835 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.710223 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:21.781179 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:21.807819 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:21.821930 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.823886 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.825796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.826483 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:21.837642 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:21.840166 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.840461 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.850384 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:21.858861 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:21.870258 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.871613 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:21.873683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:21.881984 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:21.974988 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:03:22.102402 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:22.108798 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:22.189371 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.195955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:22.196560 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:22.199546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.208088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.217245 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.235175 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.236407 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:22.238344 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.246282 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:22.246335 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:22.250486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.262316 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:22.283609 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:22.307292 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:22.347510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.347785 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.350471 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.352941 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:22.353104 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.365675 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:22.435945 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:22.442862 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:22.557839 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.562021 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.562741 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.562727 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.563983 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:22.584637 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:22.621238 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:22.653403 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:22.666211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.669533 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.670795 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.671180 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:22.672263 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.672649 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:22.701146 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:22.775748 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.775925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.778031 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:22.778103 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:22.785621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:22.788250 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.792872 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:22.884470 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.885891 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.887475 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:22.890230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:22.899812 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:22.908618 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:22.913847 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:22.922106 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:22.969218 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:23.049938 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:03:23.090713 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:23.099462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:23.101419 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:23.101886 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:23.104163 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.105553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.106889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.123971 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:23.200953 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:23.226440 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.227298 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.227481 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.228761 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:23.235930 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.237146 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.244925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.254590 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:23.259940 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:23.260193 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.262840 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:23.268680 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:23.333168 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:23.479635 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:23.595605 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.601136 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.601457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.602700 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.608217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:23.612082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.612630 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.624253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:23.630299 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:23.639559 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:23.642570 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.662196 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:23.667302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:23.718140 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:23.750020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.751528 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.752113 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.762310 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.769376 29737 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:23.769483 29737 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:23.769922 29737 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:03:23.771178 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:23.771364 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:23.771430 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:23.771615 29737 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:23.781912 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:23.783615 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:23.810583 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:23.966246 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.969221 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:23.971325 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.974381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:23.978999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:23.981009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:24.074364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.077719 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.084137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.086438 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:24.087108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:24.104444 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.108969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:24.131508 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.186609 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.187568 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.189193 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:24.196188 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.199887 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:24.211205 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:24.216975 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:24.220541 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:24.225183 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:24.249558 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:24.281324 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.295360 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.295696 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.298755 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.299252 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:24.308248 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:24.309116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:24.397255 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:24.485289 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.510088 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.510293 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:24.510787 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.511480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.513913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:24.525403 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:24.611313 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:03:24.629997 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.630103 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.633492 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.640729 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.641196 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.647591 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:24.649716 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.650712 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:24.658116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:24.661020 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:24.662528 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:24.662791 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:24.677313 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:24.679617 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:24.699708 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:24.706418 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:24.721665 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:24.959977 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:24.977246 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:03:24.980921 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:25.004530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.011184 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.012041 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:25.027730 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:25.028954 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:25.029258 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.030975 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.033605 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.038013 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.060680 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:25.068576 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:03:25.076805 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:25.080109 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.081261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:25.163290 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:03:25.172281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.172554 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.173338 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.175346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.182091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.184258 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:25.201454 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:25.246629 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:25.269191 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:25.300735 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:25.355046 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.360195 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.364435 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.368952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.371054 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.373528 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:25.482713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.484799 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.488075 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.489360 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.493901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:25.512950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.588640 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.591889 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.595921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.600171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.602044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.603113 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:25.604354 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:25.626233 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:25.636019 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:25.649217 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:25.671458 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:25.685189 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.686710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.690533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.692879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.706385 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.708583 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:25.709131 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:03:25.890089 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:25.906694 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.907609 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:25.909157 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:25.913172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:25.915951 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:25.918217 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:25.957476 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:26.017892 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:26.022330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.025836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.033438 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.034102 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:26.035169 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:26.043900 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.094808 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:26.095371 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.095985 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.096096 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:26.097645 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.104760 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:26.123996 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:26.126037 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:03:26.273110 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 02:03:26.273779 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:26.359830 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.364867 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.374528 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.379091 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.382195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:26.394680 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:26.429448 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:26.458212 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.459050 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.461876 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.465664 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.472887 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:26.481851 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:26.486020 30880 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:26.486127 30880 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:26.486496 30880 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:26.487046 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:26.487813 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:26.487879 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:26.488152 30880 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:26.491946 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:26.565117 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.567701 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.568816 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.580111 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.582870 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:26.588430 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:26.623699 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:26.687669 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.689164 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.690167 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.692312 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:26.702479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.709440 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:26.824469 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:26.871393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.874053 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.875717 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:26.886847 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:26.889591 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:26.889972 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:26.905614 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:26.931790 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:26.967571 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:03:26.989235 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:26.994830 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.004882 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.006809 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:27.012290 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:27.016422 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:27.016996 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.053875 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.057225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.061780 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.071789 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.072103 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.075449 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:27.089439 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:27.090770 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:27.108474 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:27.108530 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:27.203444 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:27.319950 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:27.328080 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.334916 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.340260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.340780 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.352268 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.352344 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:27.378437 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:27.437747 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:27.441571 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.447769 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:27.448422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.449609 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.450712 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.460487 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:27.473469 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:27.479146 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:27.510231 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.515275 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.516649 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.517913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.518985 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.525636 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:27.568898 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:27.711464 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:27.794535 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.794725 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.801241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.802264 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.817831 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:27.818321 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:27.819525 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:27.872702 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.875783 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.877005 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.880861 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.886958 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:27.887071 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:27.898823 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:27.984424 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.989797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:27.991324 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.991932 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:27.995090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:27.995838 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:28.012883 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:28.013352 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:28.020939 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:28.118695 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.119943 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:28.122584 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:28.123512 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.128811 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.144289 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:28.215494 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:28.291026 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.296933 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.298662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.302006 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:28.303529 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:28.305330 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:28.305935 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:28.392028 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:03:28.402367 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:28.413038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.414147 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.414384 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:28.415037 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.425887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:28.431257 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.476962 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.481819 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.482555 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.498462 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.499114 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:28.501618 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:28.508138 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:28.686096 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:28.726696 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.727594 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.731411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:28.732877 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.738595 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:28.739454 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:28.766796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:28.798751 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:28.848768 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.851768 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.852778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.856070 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:28.884405 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.893268 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:28.895335 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.895617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.896636 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:28.897011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:28.905207 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:28.910913 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:28.924008 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:28.929003 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.165092 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:29.218312 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.226138 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.233611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:29.235453 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.260962 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:29.262732 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.272189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.284312 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.292412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.296634 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:29.297137 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:29.303747 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.308409 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:29.314491 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.397485 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.399739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.405277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.406199 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:29.412920 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:29.433496 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:29.438553 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:29.445397 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.469998 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.508592 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:29.569159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:29.571108 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.574218 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.579676 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:29.583678 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:03:29.584275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:29.588347 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:29.593849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:29.648113 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.659416 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:29.689937 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:29.719053 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.719867 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.723897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.727123 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:29.729362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.732563 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:29.750459 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:29.756372 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:29.816915 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:29.837586 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.838083 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.838614 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.855320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:29.864370 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.866274 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:29.901906 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:29.916205 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.916621 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.919839 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:29.921522 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:29.937886 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:29.939910 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:29.981216 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:03:30.020354 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:30.110680 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:03:30.111323 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:30.159507 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.160432 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.164471 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:30.165112 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.174611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:30.179832 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:03:30.181207 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:30.272869 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:30.279359 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.283836 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.284868 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:30.285534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.293439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:30.313493 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:30.320660 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:30.321007 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.327643 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.330571 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:30.341993 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:30.348390 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:30.368753 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:30.374222 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:30.502907 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:30.526108 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:30.549386 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:30.644217 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.651077 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.658237 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:30.661984 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.662922 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:30.679951 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:30.692498 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:30.706794 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.726387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.730548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:30.733562 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.733897 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:30.738701 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:30.807857 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:30.831952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.834678 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:30.837229 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:30.845757 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.846666 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.855950 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:30.975124 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:30.984305 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:30.994182 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.995807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:30.996584 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:30.999819 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.010390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.016875 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:31.131574 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:03:31.147948 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.155148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.155412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.155748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:31.158846 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:31.169982 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:31.177417 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:31.212108 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:03:31.257547 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.260633 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.263581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:31.266126 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:31.267918 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.277138 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.358767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.358862 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.359361 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:31.366412 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.367200 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.383288 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:31.386474 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:03:31.387151 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:31.404914 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:31.409345 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:31.553673 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:31.569865 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.573225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:31.577302 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:31.577538 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.580909 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.593861 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:31.594209 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:31.686237 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:31.690317 32092 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:31.690497 32092 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:31.690935 32092 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:03:31.691603 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:31.693432 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:31.693507 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:31.693578 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:31.693871 32092 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:31.694478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.702611 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:31.708598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:31.713548 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:31.713590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.720033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.720788 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:31.724454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:31.728003 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:31.744083 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:31.759724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:31.774983 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:31.790601 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:31.851292 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:31.889655 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:31.916230 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:31.967779 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:03:32.046016 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:32.068490 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:32.090523 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.097388 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.100492 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.104241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.105440 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.105722 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.107028 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.124161 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.135915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.137239 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.146368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:32.147372 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:32.148653 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:32.148733 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:32.243845 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.244248 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.248020 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.248853 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.254633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.262531 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:32.283739 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:32.326797 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:32.343508 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:32.365309 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:32.457834 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.464808 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.471681 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.475001 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.475345 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:32.475812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:32.517949 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:32.558436 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.561430 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.562686 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.570195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.572196 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:32.572391 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:32.646931 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:32.655889 32435 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:32.656028 32435 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:32.656529 32435 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:32.657987 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:32.661804 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:03:32.661886 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:32.662098 32435 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:32.669226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.673995 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.676920 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:32.679948 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.680498 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.690018 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.779903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.787567 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.787609 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.790707 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:32.795472 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.814030 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:32.879851 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:32.960458 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:32.989465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:32.989910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.990594 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:32.991113 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.010591 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:33.011345 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:33.059885 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:33.094310 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:33.101245 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:33.135105 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.146703 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.148837 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.151970 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:33.155920 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:33.172570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.186652 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:33.196627 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:33.196892 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.204083 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:33.210769 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.211035 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.213519 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:33.325109 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:33.469370 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:03:33.477963 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:33.480894 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.483737 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.487957 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:33.493856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.497470 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:33.500584 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.561792 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.563536 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.566357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:33.570358 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.576372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.594513 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:33.642012 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:33.643519 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.644083 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.644676 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.646462 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.653853 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:33.662881 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:33.713733 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:33.878413 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:33.879205 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.880426 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:33.883915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:33.885219 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.894524 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:33.910930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:33.965749 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:33.986434 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.988029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.988334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:33.998131 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:33.999167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:33.999986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.014357 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:34.073427 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:03:34.094727 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.096024 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.097362 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.099712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:34.108135 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:34.118932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.128424 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:34.155546 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.161629 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.165338 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:34.206279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.209522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.215827 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.220955 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.222342 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.245492 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:34.275599 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.395437 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:34.433914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.435381 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.437944 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.444702 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.444712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:34.444892 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:34.463364 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:34.520550 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.560842 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:34.566718 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.570770 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.571022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.571336 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.571621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.572464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.573015 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.580258 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.586897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.606987 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:34.611898 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:34.619906 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:34.646920 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:34.662161 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:34.734699 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.790460 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.874857 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:34.878230 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:34.913767 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:34.916419 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.918196 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.918196 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.919458 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.931428 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:34.932235 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:34.988848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:34.990450 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:34.990955 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.000041 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.000312 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:35.024771 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:35.047600 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:03:35.101213 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.104276 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.106405 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:35.117079 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:35.118145 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:35.140698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:35.151166 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:35.152458 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:03:35.192363 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:03:35.239233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:35.240796 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:35.242565 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.242949 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.245391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:35.245898 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:35.250695 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:35.341320 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:35.343205 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:35.429509 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.431692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:35.434657 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.440526 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:35.441735 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.443112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:35.546546 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.547641 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.549270 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.555944 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:35.569397 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:03:35.574070 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:35.581243 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:35.585250 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:35.594334 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.603147 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.603317 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:35.608134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:35.613073 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:35.615664 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:35.623298 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:35.630972 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:35.762032 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:35.796061 1333 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:35.796165 1333 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:35.796569 1333 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:35.797513 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:35.797617 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:35.797695 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:35.797827 1333 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:35.895807 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:35.926746 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.931123 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.942549 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:35.946305 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.959290 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:35.964885 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:35.981325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.985457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.990460 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:35.991488 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:36.003717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.017901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:36.031826 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:36.093115 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.097030 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.097680 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.104902 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.104897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.109457 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:36.174336 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:36.201071 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:36.215787 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:03:36.306813 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.309098 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.310714 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.310781 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.311983 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:36.312374 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:36.320505 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:36.360452 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.406651 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.407941 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.408370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.417032 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.424865 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.426513 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:36.448546 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 02:03:36.449239 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:36.464531 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:03:36.508478 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:36.513633 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.514520 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.518023 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.518275 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.518872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.536412 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:36.585415 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.623404 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.626365 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.627599 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:36.627914 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:36.630806 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.631964 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.640537 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:36.641736 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:36.664225 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:36.810256 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:36.810501 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.830838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.835698 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.836665 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.839380 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.841480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.842061 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:36.943150 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.947616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.951665 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:36.958380 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.965452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:36.979581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:36.989284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:36.991786 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:36.992060 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:36.992486 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:36.995492 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.004192 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:37.017011 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:37.017632 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:37.018466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.020898 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:37.140568 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:37.167817 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:37.279486 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:37.331827 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.340771 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.343029 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.344122 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.345404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.354488 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:37.371793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.371901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.372692 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.376340 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.376807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:37.393623 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:37.418289 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.494740 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.499807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:37.501749 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:37.503327 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.505002 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.505975 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:37.516567 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:37.516635 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:37.691399 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:37.692148 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.695154 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.697654 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:37.699713 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.706804 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.711027 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:37.779819 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:37.787460 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:37.802251 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:37.813939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.817195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:37.817555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.817852 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.823139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.831671 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.919598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.920451 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.920715 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:37.925354 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:37.930074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:37.961990 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.028189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.028723 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.032538 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.035547 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.042208 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.054141 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:38.069123 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:38.075966 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:38.089632 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:38.222047 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:38.256258 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.258016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.262621 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:38.263912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.268821 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.269640 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.376168 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.377987 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:38.380252 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:38.385639 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.386312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.395093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.396504 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.403872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.406750 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:38.409358 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.414798 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.427579 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:38.448514 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:38.598656 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:38.669571 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:38.747283 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.747378 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.749919 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.752804 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:38.761162 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:38.770190 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:38.775840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.776098 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:38.784909 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:03:38.812739 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.813043 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.817966 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:38.823848 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.828469 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.829058 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.895597 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:38.898895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:38.900689 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.902262 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.907081 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:38.913132 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:38.967517 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:38.980268 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:03:39.001015 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:39.046242 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:39.049594 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.142395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.142566 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.143985 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:39.144244 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.153955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.170353 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:39.243798 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:39.246495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.248098 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.248379 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.249178 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.257592 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:39.265045 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:39.286224 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:39.305433 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.345932 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.350699 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:39.355424 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.356513 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.369814 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:39.370293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:39.409260 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.467041 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.469877 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:39.472045 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.476934 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:39.479270 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.483013 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.485306 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:39.502266 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:39.533291 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:39.559096 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:39.611954 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:39.626399 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:39.662772 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.663887 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.664824 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.667656 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:39.667649 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:39.677327 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.687929 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:39.786409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.792819 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.798614 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.805124 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.806167 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:39.825706 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:39.832418 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:39.841643 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.845628 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.853119 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:39.865312 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:39.871249 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:39.872807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:39.914847 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:40.019009 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:40.060175 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:40.083223 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:40.117456 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:03:40.122897 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:40.181640 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.185866 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.192077 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.207285 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:40.210613 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:40.216706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.245028 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.246239 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.250180 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.251906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:40.252527 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240503 02:03:40.254678 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:40.277202 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:40.282748 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:40.289908 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:03:40.339895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.340008 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.340181 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.345880 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.350719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:40.356213 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:40.550140 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:40.560518 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:40.567793 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.570875 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.577767 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.578058 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.586894 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:40.587546 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:40.640432 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:40.670864 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.671322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.671872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.675194 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:40.683370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:40.688333 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:40.696306 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:40.755550 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:40.778098 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.779942 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:40.786654 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:40.788852 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.792124 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.796062 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:40.854627 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:40.891201 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:40.893082 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.893610 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:40.899451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:40.904879 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:40.908298 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:40.987609 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:41.104108 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.106129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.110558 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.117506 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:41.117555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.134148 3369 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:41.134274 3369 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:41.134714 3369 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:41.135581 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:03:41.140753 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:41.147534 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:41.147636 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:41.147891 3369 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:41.206218 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:41.212610 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:03:41.219146 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.221454 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.227042 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:41.227697 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:03:41.231086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:41.231307 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.263900 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.273092 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.276262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.291522 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.297374 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:41.299899 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.300470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:41.354044 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:41.382200 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:41.427549 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:41.447014 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:41.469928 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:03:41.470541 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:41.600334 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.607517 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.616269 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:41.620935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.623807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.631191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.654381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.658485 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.659165 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.660998 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:41.663995 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:41.672251 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:41.693221 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:41.733282 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:41.734911 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:03:41.741115 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:41.771378 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.775184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.777068 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:41.777804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.778848 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.781893 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:41.883651 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:41.917510 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:41.975589 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:41.975639 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:41.978567 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.979975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:41.983933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:41.986202 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:41.996496 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:41.999151 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:42.019470 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:42.057924 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:42.093043 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.094450 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.097095 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.103917 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.116575 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:42.117933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.204934 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.209357 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.210018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.210608 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.212247 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:42.223074 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.250949 3159 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:42.251094 3159 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:42.251585 3159 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:42.252745 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:42.253000 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:42.253338 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:42.253620 3159 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:42.278540 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:42.313704 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:42.314715 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.315595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.316026 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.320513 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:42.321408 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.338737 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:42.379467 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:42.436000 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:42.518750 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:42.518885 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.520345 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.521088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.533377 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.544320 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.544498 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:42.585806 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:42.597923 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:42.636785 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.644166 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.647148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.649063 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.651652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:42.655417 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.662770 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.672188 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.674441 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:42.684254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:42.694463 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:42.708957 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:42.843253 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:42.935585 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:42.975234 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:43.018222 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.024238 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.026724 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.027060 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.028360 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.032794 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:43.040333 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.041800 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.043992 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.052073 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:43.060145 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:43.064153 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:43.086863 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:43.091596 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.174423 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.177379 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.178221 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.181576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.183944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:43.186396 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:43.190384 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:43.318914 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:43.387279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.387920 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.393244 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:43.405462 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:43.409473 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.410151 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.413813 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.498783 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.499106 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.500715 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.500970 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.512080 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.518628 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:43.587692 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:03:43.602221 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:43.602466 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:43.604089 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.604736 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.605342 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.618008 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:43.618741 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:43.697224 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:43.717407 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.718392 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.719552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:43.720748 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:43.721284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.724503 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:43.740464 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:43.763787 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:43.845878 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:43.912340 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.913354 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:43.914217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:43.916563 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:43.918275 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:43.941727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.038867 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.043292 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:44.045981 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:44.053478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:44.063035 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.064728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.081110 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:44.081327 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.088219 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:44.094588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.097271 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.108095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.116679 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:44.122371 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:44.226852 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:44.231889 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:03:44.337096 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:44.353055 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:44.396266 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:44.399384 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:44.408457 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:44.450748 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.454720 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:44.454278 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.456182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:44.463522 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.470458 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.470536 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:44.478772 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:44.479911 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:44.483006 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.485821 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.492767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.509251 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:44.529019 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:44.544479 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:44.579159 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:44.619884 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.620393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.623734 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.632185 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.632869 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:44.639379 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:44.833912 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:44.839030 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.842311 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.842381 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:44.843266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.851210 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:44.851946 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:44.921216 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:44.948329 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.949365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:44.950546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:44.959172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:44.963550 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:44.974377 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:44.988153 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:45.051622 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.055334 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.055859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.059079 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:45.062355 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:45.071187 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.078056 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:45.143147 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:03:45.166147 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.167069 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:45.170331 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:45.173620 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.178886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.191926 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:45.236549 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:45.255887 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:03:45.318256 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:03:45.352197 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:45.371201 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:45.371856 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:45.374150 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:45.375963 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.382959 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:45.385187 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.386221 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.403108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.457098 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:45.500416 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.501008 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:45.509131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:45.509344 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:45.512887 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.515652 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.515928 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.522114 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:45.524873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.540072 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:45.544974 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.545420 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.559615 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.587740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:45.694576 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:45.858192 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.858712 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:45.862434 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.871378 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.875316 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:45.886052 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.908403 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:45.929255 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:45.932567 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.934317 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:45.934880 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:45.947325 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:45.954857 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:45.972357 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:45.979735 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:45.989434 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:46.012611 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.019680 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.020370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.027948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:46.029660 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.039937 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:46.170584 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:46.284706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.286123 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.286942 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.287089 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:46.300254 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:03:46.300828 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:46.302027 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:03:46.310941 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:46.315178 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:46.367058 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.367930 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.377516 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:46.379491 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:46.380725 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:46.398478 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.400440 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:46.470363 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:46.484622 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.485989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.488972 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.489868 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:46.502511 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.514775 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:46.524874 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:46.611105 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:46.615085 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.616572 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.617334 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.618917 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.620209 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:46.634927 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:46.727866 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:03:46.802065 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.804131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.808347 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:46.809792 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:46.811417 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:46.812144 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:46.812183 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:46.813843 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.820664 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:46.840466 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:03:46.916186 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:46.916565 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.924779 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.929672 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.939451 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:46.949846 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:46.950222 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:46.951728 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.952536 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.957820 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:46.976195 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.980090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:46.985201 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.000571 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:47.054584 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:47.054868 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:47.163437 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:47.167266 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:47.304796 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.308395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:47.313938 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.316841 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.318245 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:47.319134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.329917 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.344220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:47.351706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.357698 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.366317 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.372714 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:47.449371 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:47.450933 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:47.454423 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.461513 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:47.463066 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.472260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.483351 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:47.488660 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:47.549203 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:47.574344 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:47.664986 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.666242 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.670102 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:47.675115 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.676766 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:47.688536 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:47.698491 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:47.773316 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.775357 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:47.781044 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.781870 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.787550 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.815047 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:47.892805 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.893258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.903097 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:47.904896 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:47.908071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:47.913957 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:47.920585 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:47.954206 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.013989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.014757 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:48.017395 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.019678 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.023200 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:48.031966 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.036918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:48.046381 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:48.127689 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:48.146689 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:48.159026 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.213094 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.217833 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.218566 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:48.232021 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:48.232594 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.234009 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:48.254674 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.339468 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.339731 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:48.342464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:48.342553 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.343904 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.346060 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:48.370602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.378855 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.396652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:48.402109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.416862 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:48.423148 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:48.608572 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:48.655238 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.683558 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.686290 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.693472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:48.696964 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:48.712317 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:48.713878 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:48.713958 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:48.714622 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.751185 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.754087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:48.756660 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:48.761152 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.765450 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.779291 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:48.850028 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.862541 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.862627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:48.862829 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:48.866429 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:03:48.870798 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:48.872416 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:48.886699 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:48.888020 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:49.038440 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:49.077536 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.080865 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.083194 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.083263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.091722 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:49.115535 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:49.182600 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.184626 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.187145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.192301 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:49.192981 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.208523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:49.252059 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:03:49.285938 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.292187 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.293536 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.298977 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.299376 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:49.300954 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:49.302601 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:49.308660 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:49.308986 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:49.318516 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:49.327538 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:49.406190 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:49.408793 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:49.409297 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.409891 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.410020 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.415925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:49.437963 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:49.451738 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:49.523634 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:49.552134 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:49.584014 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:49.601316 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.602205 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:49.604470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.605003 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.605907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.627390 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:49.711221 3162 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:49.711367 3162 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:03:49.711889 3162 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:03:49.713752 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:49.714082 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:49.714180 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:49.714505 3162 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:49.716966 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.720908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.729866 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.735067 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.735863 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:49.753665 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:49.779191 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:49.789170 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.791179 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.791203 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:49.791692 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:49.807992 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:49.819849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:49.924173 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:50.051596 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.055373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:50.056236 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:50.073058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.075832 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:50.086215 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.094116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:50.119802 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:03:50.124375 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:50.147383 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.148460 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.148658 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.148733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.164101 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.167245 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:50.185040 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:03:50.224472 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.229065 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:50.231562 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.233685 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.237589 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.256639 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:50.276396 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:03:50.354710 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:50.393980 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:03:50.409055 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:50.465277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.468806 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.469623 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.469918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:50.482901 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:50.495205 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.495882 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:50.541641 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:50.565646 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.567601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.567696 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.570327 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.571338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.584285 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:50.625838 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:50.670818 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.671011 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.674857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.675921 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.685698 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.700341 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:50.761407 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:50.774088 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:50.785250 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:50.790676 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:50.791204 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.796557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.804597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.806002 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:50.807739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.826961 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.987594 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.988683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:50.988911 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:50.992007 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:50.999913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:51.006105 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:51.012058 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:51.016726 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:51.104889 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.105631 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.111196 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:51.111565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:51.118316 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:51.120513 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:51.155262 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:51.168231 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.174713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.177325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:51.181286 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:51.186671 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:51.187134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.188699 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:51.217231 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:51.263049 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:51.365161 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:03:51.432020 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.434206 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.438918 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:51.441462 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:51.442615 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:51.455917 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:51.521216 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:03:51.535579 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.539801 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:51.542281 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.553005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.553236 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:51.566242 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:51.583106 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:51.588032 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:51.588871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.595902 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.598086 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.598109 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:51.602360 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:51.612425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:51.613029 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:51.861595 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:51.868378 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:51.870796 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:51.904206 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:51.913532 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:51.925287 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.925532 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:51.927829 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.931113 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:51.943472 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:51.970036 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:51.970522 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:51.972291 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.972889 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.979864 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:51.982102 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:51.990242 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:52.014268 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:03:52.014627 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:52.018549 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:52.095011 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.096565 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.096803 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.098457 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.101253 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:52.116660 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:52.120476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:52.150630 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:52.266199 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:52.287633 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.290058 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.297716 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.298476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.300276 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.318833 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.404742 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.409722 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.411242 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.412582 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.413697 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.433859 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:52.444630 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:52.445581 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:52.511951 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.513463 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.516724 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.517450 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.520619 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.522643 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.598479 6018 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:52.598613 6018 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:52.599123 6018 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:52.600642 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:52.601116 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:52.601193 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:52.601333 6018 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:03:52.621131 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.622129 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.623842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.625274 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.637305 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.648577 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:52.651721 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:52.761834 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:52.777140 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:52.834894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.836371 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.837538 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.838025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:52.840070 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.840675 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:52.851964 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:52.956779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.957016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:52.959230 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:52.960755 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.960975 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.985280 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:52.987591 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:52.999121 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:52.999677 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.003926 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.005153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.008919 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:53.010255 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:53.095467 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:53.131829 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:53.131945 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:53.204617 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:53.282878 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:53.322908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.331065 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.334475 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.338709 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.346231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:53.347477 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:53.355811 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:53.363637 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.366163 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.382504 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:53.404202 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.409999 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:53.481815 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:53.493468 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.496311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.497241 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.497422 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:53.500555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.510887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:53.571638 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:53.686966 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.688504 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.691349 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:53.699697 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:53.701231 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.702574 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.708074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:53.754252 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:53.769304 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:53.803560 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.807108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.815340 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.821350 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:53.824271 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:53.834830 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:53.878588 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:53.916822 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:53.917515 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.919203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:53.922060 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:03:53.923902 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:53.925663 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:53.935853 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:53.966043 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:03:53.983312 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:54.030272 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.030519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:54.031874 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.033798 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.039897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.052779 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:54.082585 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:54.141567 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:54.146126 6980 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:03:54.146231 6980 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:03:54.146618 6980 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:03:54.147589 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:54.147670 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:54.147730 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:03:54.147997 6475 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:54.242867 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:54.243372 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:54.248287 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:54.252264 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:54.255741 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.258335 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:54.260609 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.268703 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.269079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:54.322044 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:54.364558 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:54.365412 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.366123 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.373325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:54.373576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.392474 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:54.396967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.410400 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.414616 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.422670 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.427223 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.433899 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:54.444101 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:54.444360 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:54.466406 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:54.544560 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:54.723528 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:54.745935 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.751124 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:54.758402 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.760231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.766222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:54.772166 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:54.772665 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:54.780436 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.780869 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.804422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.804800 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.821406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.835160 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:54.911321 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.914433 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.914976 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:54.916863 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:54.917944 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:54.923292 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:54.923949 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:54.930656 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:54.967551 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:55.107057 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:55.107546 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.107549 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.107973 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.113912 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.137740 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:55.199620 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:55.232728 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.236398 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.236505 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.237737 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:55.238052 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:03:55.238229 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:55.239385 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:55.240890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:55.248282 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:03:55.260491 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:55.305191 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:03:55.326449 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:55.340060 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.342182 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.342849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:55.343082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.348290 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.355304 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:55.369045 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:55.375010 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:55.449410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.454402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.455404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.456118 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:55.459424 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.465049 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:55.472556 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:03:55.659641 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:55.659909 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.660054 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.661494 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.663522 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.668583 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:55.681859 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:55.770170 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.775780 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:55.780668 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:55.781209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.782086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:55.784803 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:55.807305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:55.814393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.815371 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.832118 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:55.836110 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:55.840628 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:55.859279 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:55.989544 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:56.000589 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:56.011543 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:56.068418 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:56.140558 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.144129 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.145435 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.151952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.157281 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.168407 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:56.191037 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:56.206465 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.206735 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.209744 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.214118 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.217619 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.246909 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:56.259322 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:56.306149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.309166 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.312934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.313901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.320346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.325116 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:56.384088 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:56.495143 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:03:56.495776 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:56.509366 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:56.509889 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:56.530860 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.535001 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.536361 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.539925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.548494 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:56.556704 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.632105 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.633850 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.634681 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.635514 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.641350 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:56.646077 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:56.657711 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:56.691813 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:03:56.715787 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:56.735369 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:56.735831 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:56.736408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:56.740870 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:56.742084 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.742096 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.742758 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:56.751566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.755765 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:56.801643 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:56.853700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.856245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:56.858480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:56.858956 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:56.863451 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:56.868975 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:56.873061 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:56.891916 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:57.058033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.058509 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:57.059329 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:03:57.060633 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:57.060830 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:57.070046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.070732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.090802 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:57.194329 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.197160 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.204545 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:57.207044 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:57.210840 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.213079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.213953 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.217851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.238122 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.238857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.239322 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:57.258126 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.263178 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:57.325496 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:57.363852 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:57.469847 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:57.567342 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.568538 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:57.570642 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:57.584239 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.585728 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.587467 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.589367 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.602614 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:57.609506 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.610749 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.618443 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:57.633664 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:03:57.642267 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.647670 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:57.684921 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:57.719570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.721457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.722208 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.722769 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.724565 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:57.736083 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:57.847718 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:03:57.920159 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:57.948323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:57.949190 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.950157 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.955307 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:57.956748 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:57.967231 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.062821 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.068514 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.073606 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.073992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.076385 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:58.076428 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:58.082094 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:58.154542 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:58.173949 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.176198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.178194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.181051 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.181334 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.183871 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:58.246501 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:58.282682 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:03:58.283522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.284025 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.288105 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:58.288713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.290642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.292861 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:58.328771 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:58.395603 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:58.498517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.498579 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.501775 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.505621 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:58.509692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.522277 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:58.565730 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:58.619160 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.620733 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.621187 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.622664 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.638744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.643321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:58.649099 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:58.649564 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:58.659554 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.671039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:58.676231 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.686277 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.700305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:58.712132 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:58.754931 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:03:58.925181 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:58.970309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.975385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:58.978256 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.988194 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:58.990772 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:58.991883 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:59.023288 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.042688 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.043694 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.044246 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.051990 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:59.054174 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.062480 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:59.071409 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.148011 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.148779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.149916 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:03:59.150094 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.154098 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.160866 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:59.164686 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:59.179442 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:59.184176 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:03:59.224501 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:59.262982 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.292546 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.293529 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.295939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.299938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.304458 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:59.304987 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.314038 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:59.359398 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:03:59.377907 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:59.439652 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:03:59.457835 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:59.463793 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.465291 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.465744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.469208 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:59.478773 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:59.488509 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.572081 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.573077 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.574312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.577603 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.587708 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:59.596932 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:03:59.611219 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:03:59.633394 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:03:59.648859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.650826 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.653465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.654134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:59.654978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.661482 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:03:59.810416 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.887919 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.891789 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.891831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:03:59.902238 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:03:59.907373 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:03:59.922642 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:59.934168 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:03:59.985579 8830 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:03:59.985718 8830 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:03:59.986176 8830 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:03:59.987094 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:03:59.988155 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:03:59.988234 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:03:59.988440 8830 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:03:59.997295 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:03:59.997377 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.998198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:03:59.999071 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:00.010211 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:00.011217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.014026 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:00.020690 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.080752 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.081250 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.085189 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.087608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.094566 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:00.109632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:00.147161 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:04:00.224776 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:04:00.240741 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:04:00.278412 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:04:00.307935 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:04:00.308648 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:00.324693 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.324759 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.324859 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.337085 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.339601 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.342110 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:00.423866 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.425988 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.433889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.434022 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.451081 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:00.461107 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:00.461356 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:00.463557 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:00.478104 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:00.519071 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:04:00.536590 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.539979 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.540208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.544869 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:00.552942 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.565454 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:00.579167 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:04:00.650997 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.651877 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.653828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.655798 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.669241 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.671876 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:00.787176 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:00.840487 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:00.860049 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.860262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:00.860857 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:00.860864 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.865299 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:00.865772 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.968603 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:00.971326 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:00.981176 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:00.986045 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:00.988797 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:00.996863 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:01.003919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.010771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.015936 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.031347 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.032402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:01.040387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.042665 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.058777 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:01.217218 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.294164 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:01.337306 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:01.339414 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.342137 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.349215 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:01.349620 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.350528 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:01.371922 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:01.398265 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.413777 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.414057 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:01.417909 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.418565 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:01.431393 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:01.472659 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.483350 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:01.494638 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:01.516610 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.519582 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.519982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.525041 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:01.526741 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.539443 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:01.561594 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.672506 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:04:01.692468 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.694533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.694818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.695887 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:01.710188 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:01.711792 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:01.712316 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:01.816432 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.834609 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:01.835206 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.835891 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.838182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:01.838646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.840085 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.852581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:01.865674 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.873346 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:01.895149 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:04:01.938966 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.939877 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:01.940304 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:01.943274 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:01.948311 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:04:01.973052 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:01.973805 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:02.021858 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:02.045339 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.045675 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.048105 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.048286 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:02.053114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.061041 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:02.074710 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:02.218071 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:02.268115 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.268786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:02.271185 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.281337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.284629 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.292039 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:02.318102 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:02.330304 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:04:02.375458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.379453 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.384876 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:02.390148 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:02.394786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.399442 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.405951 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:02.447999 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.453140 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.459790 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:02.463176 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.474114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.474815 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:02.496865 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:02.577870 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:02.609546 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:02.675472 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:02.695542 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:02.720894 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.722402 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.735597 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.737766 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:02.749272 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:04:02.762042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:02.762341 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.814095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.816097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:02.816532 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.821673 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:02.826683 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.833858 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:02.912756 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:02.917200 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.917615 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:02.922009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:02.935722 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:02.937635 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:02.963479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:03.031620 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:03.036837 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:03.071628 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.074692 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.075794 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.075976 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:03.094355 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.094623 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:03.143716 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:03.251633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.251896 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.255465 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:03.255939 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.257429 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.260304 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.362766 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.364305 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.367318 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.367453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.369089 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.372653 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:03.376965 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:03.426760 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:03.448514 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.448640 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:03.453831 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:03.454685 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.464047 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.465595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.476362 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:03.617236 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:03.661985 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:03.681563 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:03.690820 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.694092 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.696202 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.700402 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.700970 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:03.711503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.805871 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.809266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.815857 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.819841 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.826478 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:03.843736 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.874742 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.877601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:03.880544 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:03.884461 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:03.891186 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:03.895609 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:03.905723 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:03.935137 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:03.963407 10088 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:04:03.963533 10088 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:04:03.963963 10088 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:04:03.964674 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:04:03.964800 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:03.964871 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:04:03.964987 10088 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:04.058629 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:04.149964 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:04.155542 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:04.158315 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:04.160490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.167565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:04.173250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.175822 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.192414 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:04.203612 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:04.239626 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:04.248848 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.249495 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.249969 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:04.252324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.253301 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:04.272881 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:04.284493 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:04.353080 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.355643 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.359690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.361445 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:04.368878 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:04.372961 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:04.375916 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:04.385912 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:04.404285 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:04.501068 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:04.504765 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.507091 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.507124 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.513228 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:04.520632 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:04.522799 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:04:04.652189 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:04.679315 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.679523 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.681376 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.684072 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:04.685171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:04.685441 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:04.750784 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:04.805667 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.806120 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.815943 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:04.819109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.819851 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:04.825865 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:04.830066 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:04.861037 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.871901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:04.872221 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.884275 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:04.888545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:04.897315 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:04.908263 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:04.940448 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:04.958664 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:05.134812 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:05.153039 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:05.170182 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:05.174880 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.181810 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:05.183003 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.185612 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:05.186131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.192783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.242408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.247723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:05.259930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.262207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.263094 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:05.265236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:05.294970 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:04:05.335043 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.335386 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.341612 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.355527 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:05.356211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.357084 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:05.364478 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:05.474069 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:04:05.484776 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:04:05.485092 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:05.485502 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:05.554735 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:04:05.568184 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.573432 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:05.573506 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.574685 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.579906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:05.588383 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:05.599784 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:05.664897 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.669091 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.672468 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.672749 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.683887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:05.684798 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:05.685905 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:05.780254 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.781988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.785956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.790824 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:05.804492 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:05.805112 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:05.807087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.886730 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.888901 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.891209 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:05.895932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:05.896646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:05.903910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:05.904855 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:05.976601 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:06.101437 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.104398 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.108103 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.108325 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:06.116411 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:06.135020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.218699 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.236241 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.245491 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.248260 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.249013 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.249575 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.254410 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:06.266008 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:06.274273 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.278229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:06.283934 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:06.286448 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:06.302525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.307106 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:06.369765 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:06.378389 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:06.489672 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:06.494745 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:06.517493 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:06.581669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.581761 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.596042 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:06.596642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.607723 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.609966 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:06.623678 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:06.656833 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.657127 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.668853 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.678742 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:06.691545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.694686 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:06.709196 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:06.737202 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.737591 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:06.741406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.743160 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:06.743832 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:06.747403 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:06.793290 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:06.930691 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:04:06.931504 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:06.980017 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:06.992133 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:06.997776 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.003381 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.004428 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.005589 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.010824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.030817 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:07.087081 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.087129 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.088523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.089748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.096087 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.101647 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:07.109341 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:07.111613 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:07.153656 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:04:07.158951 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:07.190778 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.190989 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.194216 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.204946 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.210005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.211038 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:07.312505 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:07.324239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.328009 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.328379 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.328657 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.329818 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:07.353267 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:07.377478 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:07.378309 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:07.458514 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:07.506421 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.509229 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.512427 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.513116 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:07.514986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:07.530021 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:07.539587 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:07.546746 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:07.625414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.625726 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.629331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.631665 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:07.642074 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:07.642839 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.649624 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:07.694082 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.695569 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.699803 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:07.700933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.704051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.717249 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:07.799597 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:07.802603 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:07.911051 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:07.953155 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:04:07.979377 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:07.979947 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:07.980326 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:07.982452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:07.987650 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:07.990778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:07.992870 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:08.007681 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.076149 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.078467 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.081648 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.084127 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:08.093240 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.121207 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.139389 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.143393 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.143424 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.145020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.154310 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:08.163915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:08.273070 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:08.285281 10682 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:08.285400 10682 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:08.285812 10682 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:08.286737 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:08.286844 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:08.286922 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:08.287055 10682 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:08.309527 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:08.429397 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.436453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.436466 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.438886 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.450466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.454006 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:08.478919 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:08.483846 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:08.513579 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.516463 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.519573 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.519740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.532645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.544348 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:08.625592 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.626259 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.629220 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.631053 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.648823 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.652388 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:08.717743 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:08.739305 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:08.752022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.752393 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.754205 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.765241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.770139 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.778578 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:08.830114 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:08.849040 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:08.888545 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:08.909010 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:08.940071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.943780 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:08.944451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:08.947175 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:08.954355 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:08.955372 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.058770 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.059567 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.062453 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.064268 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:09.064416 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:09.074941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:09.116119 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:09.130815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.132525 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.132802 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.134903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.142818 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:09.146421 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:09.218819 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:09.277050 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:09.284482 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:09.295574 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:09.304762 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:09.341344 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:09.377913 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.379341 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.380892 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.381318 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.383951 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:09.408659 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:09.487964 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:09.490936 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.491858 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.494714 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.504107 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:09.509111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.523708 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:09.540194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.542227 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.552520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.552906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:09.557950 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.565230 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:09.649442 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:09.798411 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:04:09.838052 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:09.878765 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:09.883935 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.887213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.890040 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:09.890286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.890965 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.901115 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:09.903288 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:09.904266 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:09.915839 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:09.916496 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:09.922459 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:09.924038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:09.931174 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:09.953774 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:10.031955 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:10.042124 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.045967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.046422 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.054625 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.055505 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.066447 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:10.156546 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:10.161964 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:10.248519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.249215 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.249521 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.251004 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.265223 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:10.269732 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:10.348542 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:10.361986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.362844 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.363310 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.366155 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.386910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.387637 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:10.420043 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:10.447479 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:04:10.477666 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.481092 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.482746 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:10.488472 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:10.492096 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:10.492281 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:04:10.493381 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:10.494167 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:10.501989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:10.517858 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:10.581099 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.582716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.584132 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.586740 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.592353 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.592732 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:10.608613 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:04:10.625947 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:04:10.793231 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.794422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.802094 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.811184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.816200 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.822698 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:10.878541 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:10.900533 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:10.917677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.921224 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.921339 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.929050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.940830 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.948762 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:10.952132 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:10.957734 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:10.960114 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:10.964046 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:10.976107 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:10.978971 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:10.981506 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:10.987941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:11.016779 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:11.229655 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:11.292549 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.301211 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.307796 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:11.312448 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.320833 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:11.323778 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.324393 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:11.324648 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:11.339922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.341341 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.351114 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:11.353484 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:11.358758 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:11.371023 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:11.425684 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:11.466033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.471393 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.471401 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.473034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:11.475242 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.484041 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:11.497983 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:11.508996 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:11.546489 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:11.546706 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:11.660482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.662212 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.665859 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.671430 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.677852 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:11.680231 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:11.766496 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:11.780896 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.784713 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:11.787933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.798425 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.802968 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.809197 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:11.849900 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:11.894413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:11.897734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.898458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:11.898700 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:11.905081 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:11.909894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.001369 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:12.002575 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.008491 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.009840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.011336 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.019815 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:12.023931 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.033120 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:12.050678 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:04:12.066006 12399 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:12.066114 12399 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:12.066458 12399 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:04:12.067332 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:12.067435 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:12.067508 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:12.067631 12399 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:12.125123 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:12.144784 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:04:12.203197 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:12.205852 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.209432 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.209686 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.210280 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:12.232131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.246227 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.312427 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:12.333411 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.338760 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.345466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.347237 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.351858 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:12.368052 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.390333 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.392103 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.397444 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.404230 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.409477 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.420271 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:12.515745 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:12.516502 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:04:12.527053 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:12.551996 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:12.569190 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:12.655023 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:12.717782 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.719684 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.726581 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.729017 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:12.736877 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.743384 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:12.743743 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.747975 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.749177 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:12.755537 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:12.759598 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.777998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.785109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.799079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:12.800331 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:12.884703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:12.885011 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.886873 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.887161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:12.887867 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:12.895946 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:12.970393 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:13.095281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.099946 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.106626 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.108464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.110509 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.140233 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.153735 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:13.196900 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:13.204906 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.208638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.211371 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.213703 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.214088 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.225522 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.293885 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:13.315222 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.317294 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.318148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.318697 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.319082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.336139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.348906 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:13.349397 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:13.437345 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.439451 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.439890 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.440551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.441195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.461311 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.486029 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:13.626544 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.627617 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.628688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.630964 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.642923 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.646242 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.671464 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:13.727171 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:13.741421 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.747480 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.747785 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:13.749938 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.757758 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.768128 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.802472 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.807386 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.808574 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:13.809898 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:13.822765 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:13.823868 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:13.871670 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:13.888253 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:13.891201 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:13.973562 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:14.003371 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:14.079070 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:04:14.095260 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.095696 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.096673 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.103367 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.104744 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.117770 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:14.160080 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.163694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.165920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.166229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.174620 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.194639 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:14.198045 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:14.220606 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:14.253290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.257457 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.259300 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.260701 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:14.265412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.275918 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:14.289803 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:14.309223 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:14.482322 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.487125 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.487206 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.489648 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:14.496179 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.496603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.581836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.583527 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.591269 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.592617 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:14.593542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.605223 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:14.612160 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:14.626466 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:14.708703 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.708703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.710142 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:14.711212 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:14.713256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.726672 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.729465 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:14.752624 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:14.800783 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:14.806529 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:14.815871 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.823859 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:14.826354 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:14.826819 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.828784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:14.851122 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:14.966172 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:04:15.009464 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:15.024713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.024713 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.026623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.030149 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.036079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:15.037727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:15.039067 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:15.057929 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:15.153879 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:15.155642 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.156216 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.162173 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:15.166486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.166719 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.167070 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.172133 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:15.176932 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:15.199229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.207928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.211540 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.211905 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.215617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:15.228628 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:15.476881 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:15.507021 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240503 02:04:15.533293 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.533721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.545540 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:15.546036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.550213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.559399 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.562131 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.574071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.578380 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.592689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:15.594599 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:15.596153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.597354 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:15.632710 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:04:15.639370 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:04:15.682715 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:04:15.683358 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:15.690603 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:15.695798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.698804 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.699610 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.705157 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:15.707509 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:15.714620 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:15.716723 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.899094 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:15.899510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.901142 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.902452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:15.905699 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:15.911921 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:15.951227 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:15.952240 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:16.016180 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.016673 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.018602 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.020421 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:16.022094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.052065 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:16.129647 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.137313 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.139365 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:16.139555 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.140862 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.141249 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:16.176761 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:16.205356 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:16.234865 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:16.242812 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.246181 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.247134 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.248840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.256978 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:16.268533 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:16.396811 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:16.430403 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:16.456614 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.460960 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.461797 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.468281 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:16.476078 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:16.485634 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:16.488137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:16.557915 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:16.574182 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.580075 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:16.582998 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:16.583825 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.593485 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:16.597879 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.601457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:16.621675 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.635838 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.639583 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.640009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.642094 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:16.645198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:16.680316 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:16.774367 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:16.801193 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:16.832705 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:16.857937 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:16.905112 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.906841 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:16.915685 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.922730 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:16.930780 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:16.935911 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:16.972569 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:17.000244 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.000746 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.003868 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:17.004720 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.011214 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.019714 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:17.053530 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:17.071537 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:17.096823 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:17.112270 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.112571 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.113857 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:17.114408 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.114784 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.115860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:17.191510 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:17.191613 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:17.199466 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:17.231400 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.232079 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.233202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.238334 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:17.247467 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.263273 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:17.421658 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:17.427875 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.428364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.430049 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.432596 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.435190 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:17.449339 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:17.450783 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:17.462194 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:04:17.541869 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.542073 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:17.544823 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:04:17.551872 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:17.552842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.554862 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.582176 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:17.592947 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:17.593148 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.597537 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:17.599668 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.612172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:17.629794 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.635692 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:17.649884 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:17.658407 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:17.774224 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:17.809567 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:17.929711 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:17.931639 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:17.932178 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.936259 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:17.937237 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:17.951930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.953899 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:17.996655 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:17.996937 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:17.999948 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.006757 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.014051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.025936 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:18.028290 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:18.048776 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:18.103080 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.106878 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.109778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.114794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.116091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.133924 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:18.284196 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:18.318531 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.324398 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.326504 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.331310 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.332556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.334479 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:18.352603 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:18.428828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.432319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.434789 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.437193 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.439980 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:18.447741 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:18.488452 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:18.532424 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.532900 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.535063 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.535310 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.548439 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.551683 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:18.622697 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:18.647126 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.649019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.649327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.651995 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:18.654228 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.665370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:18.695991 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:18.763432 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:18.778375 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:18.812047 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:18.843636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.849375 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.850921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.852190 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.860256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.861303 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:18.973727 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:18.975116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:18.975890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:18.981772 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:18.990550 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:04:18.991230 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:18.998790 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.006115 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.012063 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.017700 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:19.022588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.024818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:19.032276 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:19.035039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.038340 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:19.151543 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:19.152308 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:19.170464 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:19.177358 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:19.185696 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:19.288868 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:19.366533 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.369541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.370842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.375968 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.396400 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.397017 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:19.401532 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.424285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.425751 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.427774 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:19.439460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.447373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:19.480829 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:19.527432 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:19.539398 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.539870 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.540258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.540612 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.552032 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.562703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:19.703533 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:19.706650 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:19.755686 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.760637 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.761337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.769860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.770627 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.775943 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:19.860781 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:19.861248 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:19.866276 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:19.869330 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:04:19.871052 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:19.871387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.871933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.878053 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.879478 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.891067 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:19.933200 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:19.961931 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:19.975605 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.980199 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:19.982134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:19.985837 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:19.988754 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:19.990348 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.089349 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.092213 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.092662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.095118 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.105230 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.119802 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:20.140851 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:20.161254 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:20.237358 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:20.258685 15481 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:04:20.258836 15481 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:04:20.259418 15481 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:04:20.260465 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:20.260759 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:04:20.260838 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:04:20.260980 15481 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:20.296257 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.302029 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.302560 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:20.303321 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.305167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:20.315399 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:20.317098 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.429000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.432591 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.436196 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.439260 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:20.450139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.452164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.452400 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.454033 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.460747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.469553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:20.471220 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.492561 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:20.529794 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:20.551903 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:04:20.568185 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240503 02:04:20.598654 10682 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:20.598835 10682 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:20.599462 10682 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:20.600534 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:20.600849 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:20.600926 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:20.601084 10682 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:20.619799 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:20.751299 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:04:20.754801 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:20.804499 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.809763 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.817337 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:20.823462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.825716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.827332 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.836364 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.837937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:20.852216 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:04:20.859126 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:20.868052 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.868393 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.870148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.972532 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:20.976089 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.977631 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:20.984180 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:20.984287 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:20.992393 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:20.992908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:21.024528 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:21.046428 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:21.093323 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:21.162281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.162281 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.163553 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.168334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:21.175974 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:21.179559 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.193552 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:21.205188 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:21.289541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.291364 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.292232 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.292837 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.300733 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:21.316802 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.401268 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.404039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.404703 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:21.406203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.412556 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.415733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:21.418633 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:21.420127 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:21.423379 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:21.458619 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:21.509552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.509802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:21.518878 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.522543 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:21.529820 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:21.542007 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.542865 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:21.645999 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:21.720217 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.720594 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.725575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:21.729336 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.740160 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.743186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:21.812534 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:21.821003 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:21.856041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.859338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.862524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:21.864004 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.869804 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.874553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.881153 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.915802 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.916900 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:21.918442 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:21.918808 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:21.936456 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:22.000458 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:22.128198 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:22.128293 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:22.155340 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:22.194276 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:04:22.211714 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.220418 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.221457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:22.222882 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.224064 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.246552 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:22.271384 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:04:22.289176 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.290182 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.290405 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:22.293227 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:22.299321 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:22.299422 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:22.299687 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.365170 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.369482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.371979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:22.376914 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:22.378700 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:22.392721 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.395120 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:22.397017 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:22.412762 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:22.485227 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:22.613494 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.613795 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.614348 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.623003 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:22.625000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.631512 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:22.641556 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:22.718536 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.719194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.719724 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.720335 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.728453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:22.738992 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:22.745231 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:22.745671 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:04:22.828714 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.832074 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.833323 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.847973 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.850476 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:22.851183 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:22.877662 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:04:22.936676 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.937649 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:22.940615 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:22.942385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:22.943887 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:22.944885 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:22.956689 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:22.968539 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:23.107331 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:23.146729 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.152596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.154057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.154078 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.154750 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.180089 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:23.279258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.282321 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.291224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.291968 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.300138 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:23.304772 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.312426 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:23.327086 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:23.331494 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.332427 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.335119 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.350629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.362808 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:23.381534 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:23.395117 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:23.493718 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:23.599110 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:23.642484 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:23.642527 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.643047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.646212 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:23.654459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.662297 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.662516 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.711648 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.722705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.725646 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:23.728309 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.728389 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.740082 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.759534 16398 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:23.759660 16398 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:23.760049 16398 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:04:23.764616 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:23.764835 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:23.764889 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:23.765178 16398 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:23.772605 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:23.796881 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.800844 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:23.801770 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.805184 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:23.821720 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:23.822629 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:23.822763 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:23.841336 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:23.850225 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:23.857867 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:23.867511 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:24.034011 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.036557 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.041122 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.041524 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.053578 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.053854 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:24.094300 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:24.099694 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:24.135923 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.138067 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.138700 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.141254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.149082 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:24.163336 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.241612 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.244032 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.250419 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.252113 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.253597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.259936 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:24.288635 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:24.352973 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:24.353966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.354961 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.367251 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.368223 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.368265 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.375294 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:24.460158 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:24.473132 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:24.559034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.561800 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.562000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.570626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:24.571190 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.580662 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.677012 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.690317 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.691114 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:24.691550 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.695278 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:24.703423 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:24.712031 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.714516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.731918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:24.733139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:24.733546 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:24.735914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:24.739956 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:24.778406 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:24.795383 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:24.826107 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:24.830104 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:04:24.848757 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:24.906771 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:24.930009 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:25.033053 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:25.058192 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:25.078542 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.081918 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.086573 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.088892 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:25.089275 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:25.089821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.091250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.102073 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.104640 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.117444 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.122447 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.126029 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.152709 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:25.177575 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:25.222801 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.226590 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.229249 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.230504 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:25.247484 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:25.250108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.257107 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:25.445930 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.449126 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.451226 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.453907 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:25.458575 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.460805 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:25.503558 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:25.551517 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.553582 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:25.556234 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.556900 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.559922 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.567061 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:25.602413 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:04:25.647684 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.651223 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.655107 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.659689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.667188 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:25.667455 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.690089 10682 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:25.690217 10682 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:25.690624 10682 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:25.691624 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:25.691866 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:25.691948 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:25.692194 10682 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:25.720853 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:25.756179 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.760448 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.763929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.765437 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:25.772029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.792316 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:25.845407 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:25.877779 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:04:25.880352 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:04:25.942417 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:25.977682 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.979020 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:25.979488 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.980654 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:25.993039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:25.997395 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:26.026860 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:26.061571 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:26.100986 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.105492 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:26.105844 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.111687 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:26.122462 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:26.124141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:26.126817 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.129281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.141146 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.158337 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:26.160168 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:26.169220 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:26.170387 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:26.180699 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:04:26.217871 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:26.259891 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:26.280259 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:26.315477 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:26.359737 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:26.468257 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.468305 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.476420 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:26.476560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:26.487430 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:26.493583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.506029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.511945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:26.514340 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:26.515492 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.525149 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.545328 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:26.546720 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:26.577584 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:26.621002 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.624536 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:26.627017 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:26.629704 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.640404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.654765 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:26.801769 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:26.844031 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:26.849797 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.850405 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.851327 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.852182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:26.854316 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:26.866523 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:26.906800 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:26.947618 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.948738 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.952057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:26.960469 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:26.974781 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:26.977483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:26.992530 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:27.014853 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.026060 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:27.053386 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.056373 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.059196 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.071040 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.072274 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:27.073890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.163084 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.163507 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.166121 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.166483 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.173920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.180203 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:27.304677 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:27.305073 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:27.308929 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.315356 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.331037 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:27.335551 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:27.367149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.371990 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.374763 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.377362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.384104 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.398102 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:27.474356 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:27.497069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.502242 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.504334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:27.506875 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:04:27.518170 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.524152 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.525091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:27.532070 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:27.535086 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:27.535547 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.536824 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.546938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.546933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.549645 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:27.569831 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.576224 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.687426 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:27.696913 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:27.871964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.874562 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.880239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.889106 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.895102 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:27.895817 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.906949 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:27.913200 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:27.915925 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:27.916174 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:27.929518 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:27.930346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:27.962532 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240503 02:04:27.964144 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.041990 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.048524 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.049432 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.050498 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.052345 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.070268 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:28.224198 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:28.247493 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.247684 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.252095 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.262841 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.266252 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.272071 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:28.320816 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:28.364121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.364531 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.365231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.365397 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.368160 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.387077 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:28.447464 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:28.468326 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.470132 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.474144 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.482820 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.488004 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.490793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:28.552487 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:04:28.572768 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.572850 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.574990 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.584542 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.586074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.599716 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:28.609172 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:28.623203 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:28.640623 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:28.710407 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:28.760823 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:28.791095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.792937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:28.793110 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:28.802888 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.802963 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.806303 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.812433 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:28.828652 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:28.901547 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:28.920609 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:28.922439 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:28.924638 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.924773 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:28.930025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.930685 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.938966 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.943131 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:28.944125 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:28.947387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:28.952620 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:28.969906 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.998416 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:28.999061 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:29.151835 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:29.229658 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:29.298636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.304759 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.309451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.314977 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:29.320894 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.339176 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:29.353078 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.353497 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:29.367718 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.376158 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.383453 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.385907 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:29.397406 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:29.438866 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:29.457947 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.460135 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.465612 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:29.471221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.472766 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.481209 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:29.530207 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:29.622485 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:29.685971 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.690017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.691884 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.693727 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:29.695518 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.701658 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:29.789978 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.791409 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.791649 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.796173 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:29.800773 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.817854 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:29.839181 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:29.884248 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:29.904181 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:29.905073 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:29.906212 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.906226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:29.907567 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:29.920953 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:29.924364 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:29.936595 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:04:29.950873 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:29.981737 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:30.011667 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:30.012446 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:30.015784 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.017660 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.018106 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.020663 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:30.022357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:30.030656 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:30.039485 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:30.113675 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:30.235953 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.237836 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.238493 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.239904 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:30.241923 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:30.244319 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:30.253059 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:30.350499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.356961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:30.357203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.358084 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:30.359551 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:30.362190 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:30.382642 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:30.385691 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.387307 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:30.400662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.413154 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:30.425155 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:30.427598 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:30.437913 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:30.461167 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:30.608922 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240503 02:04:30.744686 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.745405 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.746213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.750417 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.758723 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:30.761459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:30.772265 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.776938 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:30.780570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.782783 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:30.807575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:30.814327 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:30.814792 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:30.890044 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:30.900876 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:04:30.904455 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:30.904768 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.905606 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.907370 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:30.908628 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:30.918886 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:30.928689 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:30.928892 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:04:30.929876 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:30.996667 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:31.121393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.123834 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.124421 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.125708 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:31.132036 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:31.134490 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:31.135670 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.155714 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:31.218811 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:04:31.229583 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.232605 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.234764 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.235888 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.246449 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.256347 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:31.303828 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:31.334429 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.336311 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.340219 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.340906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:31.346625 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.362104 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:31.371423 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:31.387955 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:31.452255 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.453776 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.457221 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:31.458927 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:31.465764 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:31.466073 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.485225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:31.505416 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:31.649406 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.652981 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.653194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.653239 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.659926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.670197 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:31.677570 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:31.688313 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:31.772318 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.778396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.780162 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.780968 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.783255 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:31.826169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.839957 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.842043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:31.845511 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:31.846387 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:31.850438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:31.868083 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:31.885866 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:31.973685 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:32.028488 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:32.052873 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:32.131294 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.134238 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.135221 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.137142 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:32.141510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.149255 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.205664 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.210727 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.216711 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.227043 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.227818 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:32.237488 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.287320 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.288565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.292833 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.296823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:32.304476 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:32.307600 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.319799 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:32.346578 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:32.359125 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:32.395404 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:04:32.443953 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:32.453455 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:32.474320 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:32.514659 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:32.530452 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.533782 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.538061 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:32.546197 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:32.550668 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:32.554150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.556157 19056 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:32.556273 19056 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:32.556645 19056 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:04:32.559927 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:32.560842 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:32.561288 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:32.561350 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:32.561640 19056 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:32.565537 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:04:32.631894 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.634529 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.635380 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.637547 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.637641 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.648072 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:32.668656 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:32.735400 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.740614 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.745263 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.748888 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.752174 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.767949 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:32.860221 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.863636 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:32.864502 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:32.867301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:32.873759 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:32.877164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:32.885892 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:32.981506 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:33.039194 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:33.039882 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:04:33.040503 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:33.064297 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.064607 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.075796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.076081 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.091610 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:33.094798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.193213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.204756 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.209072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.213819 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.235004 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:33.240729 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.251600 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.254328 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.260275 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.276918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.289858 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:33.295141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:33.313966 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:33.357357 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:33.506618 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:33.570335 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:33.590612 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.592155 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.594031 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.610736 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.611907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:33.614818 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:04:33.614894 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:33.623406 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:33.625855 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.637488 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:33.643946 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.648511 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.653337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.658756 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.668792 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:33.757915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.761149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.763160 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.778124 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.782966 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.785248 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:33.789953 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:33.797572 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:33.810685 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:33.869076 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:33.943611 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.954285 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:33.958850 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.963130 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:33.964246 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:33.988188 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:33.996666 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:34.079061 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.081568 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:34.084133 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.093509 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:34.096871 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:34.110062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:34.156738 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:34.186018 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.187683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:34.189479 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.197777 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:34.218020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.220026 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:34.230247 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:34.244307 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:34.293967 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.296270 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:34.299108 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.307860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:34.315342 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.315575 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:34.328519 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:34.346951 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:34.506399 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.515812 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.517926 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:34.518139 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.524621 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:34.529556 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:34.529881 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:34.540155 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:34.606420 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:34.623570 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:34.623991 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.635623 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:34.643121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.643121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.658124 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:34.667048 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.676290 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.679319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:34.680981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:34.691668 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:34.693962 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:34.727444 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:34.838943 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:34.899832 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:34.939008 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:34.947541 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:34.963322 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:35.014348 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.022461 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.023921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:35.025889 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:35.027148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.028368 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:35.036231 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:35.040079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.066027 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.070086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.074009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.075191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.076730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.091933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.183017 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.183709 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.187009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.187053 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.191928 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.193080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:35.241837 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:35.243088 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:04:35.381567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.382267 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.382966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.385178 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.385809 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.393843 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:35.498152 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:35.499243 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.501747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.502461 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.503407 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.505476 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.515213 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:35.580178 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:35.593961 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:35.601943 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.607105 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.607892 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.608162 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.608471 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:35.624595 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:04:35.625432 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:35.634943 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.699064 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.702231 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.704183 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.704568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.708235 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:35.723248 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:35.733668 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:35.900296 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:35.925383 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:35.925814 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.926206 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:35.926957 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:35.927300 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:35.940191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.000196 20106 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:04:36.000346 20106 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:04:36.000916 20106 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:04:36.001657 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:04:36.001696 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:04:36.002585 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:36.002667 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:04:36.004290 20106 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:36.052219 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.053830 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.061424 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.063776 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.064200 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.067122 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:36.075281 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:36.079919 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:36.087684 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:36.101096 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.111228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.113649 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.115695 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.127540 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 02:04:36.129740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:36.189352 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:36.239261 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:04:36.336979 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:36.381603 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:36.420750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.421275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.422322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.424813 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.424849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.427381 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:36.436728 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:36.488982 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.492159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.492564 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:36.493209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.497097 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.498849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.513305 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:36.518077 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:36.582414 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:36.597612 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.597940 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.602505 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:36.607957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.610293 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.611961 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.661965 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:36.710423 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:36.737581 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.739856 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.741171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.744724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:36.752532 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.754694 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.778733 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:36.863235 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:36.908236 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.917114 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.919226 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:36.920356 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:36.928480 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:36.939329 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:36.940799 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:36.975533 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:37.029284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.035974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.037055 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.037775 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.046173 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:37.046402 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:37.058661 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:37.089181 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:37.104455 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.107168 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.107863 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.117810 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.120317 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.132450 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:37.271010 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:37.282470 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:37.358018 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:37.364468 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.368422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:37.371521 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:37.372357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.374857 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.380321 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.400166 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:37.450745 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:37.473639 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.475284 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.478673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.479319 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.483011 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:37.487912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.534407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.534888 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:37.536510 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:37.540794 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.541425 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.543622 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.560921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:37.613523 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:37.620765 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:37.773828 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:04:37.816082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.819517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.830922 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.832036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.842543 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:37.855099 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:37.878561 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:37.899999 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.901400 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.910655 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:37.913498 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:37.915257 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:37.922509 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.005189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.006693 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.012876 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.014470 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.017763 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.017808 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:38.061693 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:38.062278 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:38.139935 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:38.145161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.145453 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.146186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.150727 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.151612 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.151810 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:38.164484 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:38.217859 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:38.298382 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:04:38.330245 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.333899 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.336186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.338296 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.339298 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.353386 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.445464 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.446228 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.455919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.455933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.457991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.490211 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:38.490614 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:38.508437 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.509660 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.511191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.513708 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.521605 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.541431 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:38.550947 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:38.707491 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:38.714342 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:38.717679 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:38.802433 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:38.812783 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.814407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.818089 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.821036 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.831614 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.844184 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:38.866515 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:38.888229 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.891444 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.891644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.902472 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:38.918483 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:38.918493 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.922376 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.970448 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.973277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:38.976495 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:38.983992 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:38.994120 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:38.999441 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:39.031519 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:39.035529 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:39.128696 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:39.216109 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:39.216131 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.216862 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.223945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.230165 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:39.232210 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.250866 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:39.318691 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:39.327646 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.328120 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.329878 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.330224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.334101 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:39.344307 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.430262 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.430439 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.436787 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.437774 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:39.445317 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:39.452136 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.452473 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:39.466885 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:39.474170 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:39.508213 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:39.524058 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:39.544090 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.553340 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.553602 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.559242 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.560134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.568270 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:39.748422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.748788 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.754523 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.754891 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.757354 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:39.768587 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:39.787456 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:39.818017 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:39.865950 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:39.888870 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.889379 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.896435 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.896884 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.898447 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:39.898885 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:39.916101 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.917416 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.926338 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:39.936031 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:39.936213 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:39.942699 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:39.976423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:40.052929 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:40.077412 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:40.233445 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:40.239647 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:40.270372 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.271394 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.279315 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.282689 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.289906 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.290616 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:40.314119 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.330721 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:40.330747 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.332032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.350895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.355737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:40.365605 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:04:40.399055 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:40.444779 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:40.445909 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.448515 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.451383 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.454851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.460137 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.464766 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:40.648130 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.649364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:40.649827 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 02:04:40.652094 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:40.652782 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:40.660395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.660737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.662796 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.760054 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.762996 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.765226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.766079 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:40.773918 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.781230 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:40.789880 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:40.832273 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:40.870591 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.878741 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.878810 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.879470 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.888489 21718 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:40.888607 21718 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:40.888991 21718 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:04:40.889197 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:40.892500 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:40.892669 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:40.892766 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:40.892905 21718 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:40.904412 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:40.922478 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:40.970516 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:04:40.978632 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.982234 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:40.988034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:40.988704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:40.989156 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:40.989574 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:41.033438 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:41.140482 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:04:41.141129 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:41.160214 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:04:41.191105 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.191309 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.200170 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.202406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.212366 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.214413 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:41.232590 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:41.242419 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:41.312171 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:41.321162 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:41.323330 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.326207 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.326299 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.326543 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.331975 10682 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:41.332083 10682 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:41.332489 10682 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:41.333756 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:41.336351 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:41.341001 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:41.341080 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:41.341212 10682 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:41.342181 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:41.349996 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.361128 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.370255 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.373564 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.375880 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.376799 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.404587 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:41.507943 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:41.689889 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:41.706506 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.711154 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.714051 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.717602 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.727376 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:41.728905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.730185 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:41.731927 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:41.738509 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.759577 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.763748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.771695 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.782464 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:41.795593 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:41.831022 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:41.837405 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:41.873409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.874238 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:41.874737 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:41.878294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.878523 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:41.903213 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:42.012418 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:42.051847 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:42.056454 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.065598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.065965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.068203 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.069731 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:42.093952 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.183429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.184969 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.187278 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.194958 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.201170 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.204501 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:42.207849 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:42.213007 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:42.220714 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:42.252808 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:42.253000 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:42.264117 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:42.299839 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.301363 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.301399 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.301818 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.309218 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.311380 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:42.372985 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:42.395489 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.396250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.400595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.404553 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.414153 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.434051 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:42.621868 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.627283 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.630996 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:42.635887 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.637537 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:42.637843 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:42.653102 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:42.657325 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:42.730754 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:42.737218 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.737928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.739621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.740491 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.747953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:42.748581 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:42.752229 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:42.802821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:42.803278 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:04:42.804172 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.805531 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.806707 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:42.820578 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:42.823565 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:42.857784 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:04:43.066466 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:43.092267 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:04:43.105782 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:04:43.133594 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:43.136750 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.136802 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.144814 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.147364 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:43.159276 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.159482 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.167081 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.187048 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:43.189618 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:43.190042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.191110 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.204944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:43.221917 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:43.308097 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.312629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.313110 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.314155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.318084 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:43.324796 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:43.325192 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:43.463023 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:43.494786 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:43.496977 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.500129 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.501345 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:43.508005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.509553 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.516709 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:43.582798 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:43.625016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.627530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.631042 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.631141 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:43.634995 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:43.650733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:43.714429 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:43.739809 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.740284 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.745411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.745508 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.748293 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:43.761737 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:43.819272 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:43.841687 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:43.841778 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:43.849368 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:43.850394 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:43.858278 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:43.868119 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:43.871042 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:43.878080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:43.971367 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.010252 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.042083 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:44.060565 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.064411 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.066831 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.074437 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.076130 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:44.088699 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:44.146636 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:44.186656 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.187474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.195024 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.199640 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:44.213539 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:44.214004 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.236083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.239008 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.253021 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.255308 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:44.260756 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.265118 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:44.296870 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.335386 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:44.420322 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.488003 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:44.523710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.529814 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:44.530117 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:44.531338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.532106 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.533757 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.603688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.607425 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.608245 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.610689 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.612661 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:44.617616 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:44.639160 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:44.708415 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.711989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.714664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:44.715677 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:44.717288 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:44.719558 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.725212 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:44.740558 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.853883 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:44.856549 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.858455 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:44.864596 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:44.871476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:44.871529 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:44.884627 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:44.902336 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:44.906818 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:44.908327 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:44.912623 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:45.027217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.028403 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.030153 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.031253 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.040809 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:45.043939 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:45.118683 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:45.137293 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.140108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.142267 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.143842 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:45.164871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:45.166904 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:45.171860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:45.223057 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:04:45.228864 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.229476 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.232328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.232510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.243327 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:45.247596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:45.256479 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:45.290897 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:45.465188 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.465770 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.466826 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.468046 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:45.468776 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:45.473212 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:45.479274 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:45.493067 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:45.584961 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.590531 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.593029 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:45.608649 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:45.610572 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.617386 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.623165 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.626344 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.639271 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:45.639665 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.658071 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:45.665357 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:45.695235 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:04:45.718751 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:45.752961 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:45.798998 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:45.855726 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:45.874737 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:04:45.960153 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.961540 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:45.969113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:45.984841 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:45.995661 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.004086 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:46.005301 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.008803 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.012336 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.017829 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.018893 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.046432 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:46.136761 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.136781 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.139365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.140486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.140630 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:46.143271 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:04:46.143944 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:46.182840 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:46.188231 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:46.200991 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:04:46.237581 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:46.281250 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:46.325393 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:46.330030 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.332840 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.339296 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:46.345507 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.346623 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.350811 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:46.426805 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:46.452641 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.456115 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.458652 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:46.461335 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:46.461757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.463080 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:46.463812 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.562666 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.564739 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.565888 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.567016 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.576027 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:46.580675 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:46.589511 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:46.666213 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.669847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.670732 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:46.678066 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:46.679224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:46.679376 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.680852 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:46.749207 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:46.811123 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:46.882685 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.883590 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.885866 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:46.887079 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:46.891804 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:46.895969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:46.908983 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:46.941258 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:47.004570 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:47.012778 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.015909 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.018041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.023074 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.026505 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:47.030917 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.034072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.037626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.044526 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:47.065297 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:47.076957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.081626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.084770 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:47.148869 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:47.158543 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:47.314271 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:47.364414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.368319 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.369565 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.376026 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.376641 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.391944 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:47.445883 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.446971 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.450505 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:47.451994 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.468334 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:47.476768 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:47.477921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:47.517925 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:04:47.518903 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:47.556895 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:47.560806 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.561429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.562515 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.563797 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.564743 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:47.576222 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:47.614461 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:47.641461 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:47.704440 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.705253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.706988 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.715631 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.719306 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.740543 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:47.743678 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:47.783592 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:47.872539 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.874106 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:47.875468 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:47.877152 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.889940 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.896502 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:47.942449 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:47.943307 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:47.974882 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:47.985121 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:47.987298 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.989341 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:47.993723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:47.994498 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:48.006901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.018406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:48.046193 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:48.075834 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.080816 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.082880 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:48.088444 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.098795 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:48.110015 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:48.188004 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:48.192849 23423 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:48.192981 23423 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:48.193481 23423 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:04:48.194653 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:48.194839 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:48.194898 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:48.195039 23423 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:48.285977 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:04:48.321033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.322535 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.323581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.328320 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:48.334211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.341802 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:48.374687 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:48.424862 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:48.442291 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.447839 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.454705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.472062 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.474610 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.479439 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:48.484395 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.494400 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:48.494598 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:48.502146 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:48.505126 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:48.506713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.518067 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:48.769328 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:48.818522 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:48.820960 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:48.849668 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.850378 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:48.850836 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.856971 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:48.857596 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.868906 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.869963 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.877976 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:48.883347 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.892457 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:48.895702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:48.901955 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:48.915947 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:48.941370 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:04:49.002301 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.006253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:49.006619 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:49.007128 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.012617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.013520 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.014348 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:49.179244 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:49.203828 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:49.214222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.216475 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.217062 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.217875 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.223955 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.227958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:49.288762 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:49.304023 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:49.316084 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:49.323827 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.325081 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.327322 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.327908 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.328859 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.333256 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:49.349087 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:49.378121 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:49.435667 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.442899 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.446134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.446835 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.449919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.455946 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:49.546711 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.547740 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.550938 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.562438 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.563076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.563222 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:49.566411 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:49.630573 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:49.757140 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.758098 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.758503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.767838 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:49.770932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.772639 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:49.773002 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:49.786058 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:49.836650 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:49.837409 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:49.880631 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:49.882725 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:49.888909 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.890385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.893185 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:49.906842 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.908677 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.925027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:49.930591 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.949188 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.951153 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:49.964231 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:49.968765 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:49.993175 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:50.054612 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:04:50.060772 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:50.199139 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:50.263710 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:50.264523 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.264575 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.266088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:50.269018 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.284910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.288394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:50.308815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.318730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:50.319274 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.329284 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:50.331670 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:50.342680 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.347878 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:50.412750 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:50.438408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.439167 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.441159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:50.444686 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.451587 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.459458 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:50.512156 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:50.627506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:50.631974 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.633483 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.635926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.640998 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.650136 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:50.676985 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:50.749233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.749468 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.749873 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:50.752250 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:50.756074 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:50.757704 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:04:50.758296 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:50.760736 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:50.775920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.861989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.866236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.870054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:50.876004 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.880889 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.888530 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:50.949280 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:50.961352 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.967315 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:50.968969 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:50.970777 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:50.987634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:50.990803 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:50.993510 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:51.019340 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:51.151293 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:51.175019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.175318 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.184334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:51.185580 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.196899 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:51.196916 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:51.216953 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:51.280838 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:04:51.281615 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.295017 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.303318 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:51.306766 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.310886 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.311811 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:51.312443 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:04:51.315379 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.330667 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.333442 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:51.341547 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:51.347640 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.354283 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.354557 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:51.368575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:51.416422 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.427124 24366 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:51.427266 24366 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:51.427676 24366 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:51.428767 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:51.428938 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:51.429004 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:51.429142 24366 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:51.440327 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:04:51.635419 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.646525 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.648022 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.648098 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:51.648308 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:51.648543 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.650475 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:51.714378 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.716014 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.721846 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.723054 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:51.723951 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:51.729822 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.730953 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:51.732774 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:51.822124 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:51.823194 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:51.825174 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.833621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:51.833860 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:51.840716 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.849042 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:51.863376 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:51.957455 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.960243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:51.963465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:51.968062 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.978602 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:51.984097 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:52.037670 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:52.049803 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:52.137305 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.137455 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.142179 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:52.148371 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.150646 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.157071 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:52.246273 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:52.248533 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.261600 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:52.261911 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.262743 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.263921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.266043 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:52.292704 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:52.304967 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:52.313270 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:52.329438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.330001 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.331518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.331641 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:52.331828 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:52.340596 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:52.346078 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:52.413925 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:52.428823 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:52.487676 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:52.574905 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:52.574923 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.587893 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:52.589818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.596401 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.598551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:52.654248 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:04:52.666728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.666788 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.672673 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.674124 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:52.678198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.693107 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:52.743571 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.744844 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.750861 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.753088 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:52.754539 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.761157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:52.820029 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:52.906301 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:52.907873 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:52.915416 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:52.982017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:52.985926 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.988420 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:52.996268 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:52.999055 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:04:53.010874 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.012300 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.101480 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.104286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.107875 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.108605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:53.115645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.117157 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.203729 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.206064 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.209137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.210234 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:53.213835 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.215282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.269491 25134 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:53.269616 25134 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:04:53.270030 25134 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:04:53.271157 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:53.271242 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:53.271296 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:53.271431 25134 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:04:53.271991 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:53.307200 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:04:53.307886 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:53.322206 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.322239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.326309 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.333892 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.334146 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:53.352960 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:53.354357 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:53.372282 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:53.444347 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:53.520447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.522284 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.522722 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.542811 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:53.542928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.544054 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:53.571707 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:53.584384 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:53.653520 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.655052 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.658702 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.658936 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.665939 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.669381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.669575 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:53.672163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:53.677350 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:53.681771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:53.688186 24013 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:04:53.688316 24013 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:04:53.690948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:53.692631 24013 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:04:53.693989 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:53.694578 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:04:53.694645 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:04:53.694936 24013 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:53.704274 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:53.727957 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:53.767447 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:53.769655 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:53.793072 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:53.979749 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:54.022603 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.032433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.034209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.035125 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.038635 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.052882 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:54.055954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.065217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.065852 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.074318 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.088691 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.092573 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.149188 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:54.192687 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.193290 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.197352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.199326 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.205749 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.207998 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:54.216850 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:54.226800 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:54.326414 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:04:54.330149 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:54.337996 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:54.370682 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:54.404645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.405458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.406289 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.409279 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.411645 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.418931 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.431553 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:54.468164 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:54.480314 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:54.514573 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.515388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.517186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.520432 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.522475 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:54.528299 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.618813 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.619597 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.625116 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.626050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.632525 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.648463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.653741 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:54.728976 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.733052 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.733258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.733486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:54.734437 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.756280 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.773295 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:54.894019 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:54.940691 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:54.950995 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:54.954113 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:04:54.954226 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:54.955664 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.959748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:54.959998 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:54.963955 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:54.964592 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.080255 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.080552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.084084 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:55.084331 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.088063 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.099088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:55.123490 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:55.124053 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.127796 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.139463 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:55.148357 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.148669 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:55.150274 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:55.205008 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:55.225731 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:55.358678 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:55.420272 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.425060 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.431337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:55.439422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.443220 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.453076 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:55.473992 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:55.500846 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.500947 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.503643 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.509135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:55.518471 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.531471 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:55.601548 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:55.608309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.609264 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.611114 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.612360 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.616325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:55.628039 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:55.672796 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:55.744688 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.747014 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.748948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.749802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:55.753662 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:55.754727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:55.764228 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:04:55.839996 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:55.860800 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:55.915315 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.918463 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.921129 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:55.921540 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:55.933784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:55.953748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:55.989264 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:56.042274 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.043993 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.047703 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.050098 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:56.068220 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.080318 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.119890 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.120487 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.120546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.121634 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.125306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.139928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:56.281492 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.283536 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:04:56.284140 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.362419 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:04:56.368252 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:56.373560 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.376873 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.377769 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.388543 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:56.388702 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.397148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:56.397549 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:56.418576 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.473541 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:56.483471 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.489010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.489686 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.491667 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:56.493583 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.500627 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:56.531081 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:04:56.551646 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.552436 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.562736 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.563033 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.563643 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.565348 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:56.604624 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:56.649114 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.751317 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.836392 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:56.838415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.839483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.841434 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:56.842919 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.854430 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.859910 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:56.876331 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:56.922453 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.922742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:56.922940 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:56.924521 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:56.925629 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:56.931914 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:56.935446 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:56.955601 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:56.981148 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:57.003101 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:57.031298 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.031996 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.032631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.033124 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.037518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.056109 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:57.086117 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:57.145823 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.147859 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.151285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.153807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.156641 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:57.178927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:57.204053 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:57.273790 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:57.346565 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.349261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.350994 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.353394 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.353626 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:57.364403 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:57.369916 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:57.430145 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:57.477182 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.483273 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.493069 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.494573 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.494731 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:57.499890 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:57.500965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.508901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.512099 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.520504 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.521086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.546056 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:57.554208 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:57.612519 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:04:57.688966 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:57.776383 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:57.834439 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:04:57.843770 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:57.845618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.847935 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.871856 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.872359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:57.873996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:57.878417 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.885445 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.886363 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:57.894973 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.911665 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:57.925856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:57.926393 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:57.960286 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:58.021081 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.021595 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.027374 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.027814 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.028985 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.041724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:58.051795 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:58.119612 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:58.143337 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:58.205847 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:58.233494 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.238581 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.239043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.239148 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.246235 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.260792 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:58.283663 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:58.341413 24366 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:04:58.341552 24366 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:04:58.342008 24366 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:04:58.342546 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:04:58.343446 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:58.344188 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:58.344259 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:04:58.344532 24366 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:04:58.347404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.352046 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:58.353235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.354552 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.355988 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.359928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.459353 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.460374 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.461089 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.463133 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.463303 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:58.467222 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:58.474265 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:58.543154 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:58.567997 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.570195 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.574291 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.587247 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.587368 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:58.592377 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:58.691205 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:58.758512 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:58.774075 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.774132 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.785821 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.787235 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:58.787667 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.788419 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:58.839689 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:04:58.868481 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:58.901633 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.902369 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.903793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.913902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.915323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:04:58.919090 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:58.920233 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.920840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:58.925942 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:58.934221 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:58.939275 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:58.939927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:58.958315 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:58.968205 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:59.040920 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:59.058916 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:04:59.271733 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.275743 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.278548 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:59.288369 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:59.289304 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.300912 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:59.314100 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:59.325513 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.331952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.333887 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.340220 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:59.341104 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.342098 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:59.430253 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:59.432217 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.436875 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.439919 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:59.441830 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:59.452540 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:59.456070 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.459722 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:59.478384 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:04:59.488622 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:59.493714 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:04:59.499159 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:59.641876 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.646620 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.650831 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.650843 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:59.653728 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:04:59.654724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:59.656720 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:59.706625 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:04:59.747906 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.751806 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.754731 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:59.756598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.756850 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.769606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:04:59.781615 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:04:59.821952 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:59.870175 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.870254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:04:59.870981 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:59.872400 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.873724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:04:59.894517 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:04:59.946148 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:04:59.975894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.980235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.983533 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:04:59.985087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:04:59.985689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:04:59.988122 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:04:59.996654 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:00.048132 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:05:00.099730 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:00.179493 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.182112 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:00.183445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.187932 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:00.188694 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:00.190905 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:00.192101 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:00.294173 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:00.309576 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.310082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.317566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.319692 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:00.321437 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:00.322325 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.330018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:00.338901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.341024 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.341264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:00.352660 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:05:00.352973 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:00.364399 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:00.542723 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:00.671072 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:00.688766 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.691601 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.695910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:00.696036 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.705508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:00.705943 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:00.708405 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:00.710469 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.713449 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.725036 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:00.759642 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:00.765215 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:00.817548 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:00.818588 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:05:00.819264 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:00.835202 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.835541 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:00.841010 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:00.841665 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:00.842319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:00.849032 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:00.850534 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:00.893990 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:05:01.011579 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:01.056321 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.061203 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:01.061942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.074931 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:01.075872 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.076177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:01.119817 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:05:01.162348 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.167438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.170626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:01.171000 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.180107 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:01.191802 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:01.235126 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:01.268848 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.273787 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.275642 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.275810 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:01.282852 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.288676 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:01.321365 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:05:01.325927 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:01.376667 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.377144 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.378150 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.380311 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:01.389381 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:01.405925 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:05:01.409399 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:01.452243 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:01.456307 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:05:01.460438 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:01.526466 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:01.552945 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:01.581744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.583737 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.584537 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.590914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:01.595508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.600874 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:01.645140 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:01.683204 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:01.700973 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.703408 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:01.705916 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.716097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.716879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.717808 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:01.734645 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.739394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:01.746084 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.749606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:01.752815 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:01.757737 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:01.758984 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:01.930045 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:01.983326 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:02.072273 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:02.074661 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.081233 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.087916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.101706 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.102494 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.111415 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.112411 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.117411 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:02.120208 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.126871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.139202 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:02.157018 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:02.211671 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:02.218935 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:02.237805 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:02.242416 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.245756 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.246055 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.248761 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.249265 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.268105 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:02.306375 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:05:02.387607 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:02.423818 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:02.434325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.436425 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:02.440227 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:02.441032 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.445842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.448330 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.462188 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:02.491864 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:02.555084 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.556128 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.560220 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:02.572471 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.573284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.574159 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:02.646127 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:02.648175 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:02.661962 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.662984 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:02.664229 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:02.666589 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.667915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.676141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:02.685145 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:02.685959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:02.772872 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.779875 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.781541 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.783496 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.788337 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.807279 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:02.962363 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:02.979178 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.984011 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:02.987915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:02.993304 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:02.999190 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:02.999961 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:03.069656 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:03.107096 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:05:03.109865 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.122815 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:03.125546 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.125830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:03.126605 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.128178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:03.129788 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:03.137173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:03.146075 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:03.147547 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:03.163097 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.164367 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.175231 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.193027 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:03.237713 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:03.449963 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:05:03.481937 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.484838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.487588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:03.492451 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.494240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:03.503736 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:03.509948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:03.554394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.555361 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.557018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:03.558192 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.562160 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:03.567720 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:03.568087 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:03.591153 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:03.647096 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:03.654394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:03.654843 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.656230 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.660358 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:03.666124 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:03.679018 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:03.723659 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:03.804440 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:03.811018 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:03.811745 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.813927 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.815346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.818758 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:03.833076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:03.852232 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:05:03.912201 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:03.973614 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.975106 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.977824 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:03.978565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:03.980535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:03.984639 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:04.037333 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:04.068840 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:04.092638 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.093039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.093909 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.094223 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:04.103443 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:04.119971 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:04.124828 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:04.144484 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:04.172199 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.172201 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.174520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.176474 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:04.178105 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:04.179946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:04.363124 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:04.415283 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.418458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.419456 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.430415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.432548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:04.454715 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:04.532266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.535713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.535845 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.541452 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.547011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:04.570349 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:04.570911 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:04.576406 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:04.583328 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:04.588277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.594339 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.595542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.606422 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:04.608899 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.624881 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:04.704468 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:04.747354 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:04.805171 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:04.835690 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:04.886585 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.894546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.895028 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.895473 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.897322 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:04.901908 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:04.907590 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:04.913035 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:04.913507 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:05:04.961588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.961934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:04.965325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:04.970163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:04.976222 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:04.984302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:05.012444 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:05.092626 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.093761 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.099036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.099529 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.105875 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:05.116667 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:05.137960 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:05.147084 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:05.160192 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:05.185906 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:05:05.221328 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.223366 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.234939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.235585 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:05.235920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.253737 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.409315 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.411285 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.412618 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.417790 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.428682 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.431910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:05.492365 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:05.506937 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:05.529203 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.532184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.534907 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:05.540264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.542263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.549042 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:05.576400 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:05.595083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.599524 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.605509 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.605723 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.608326 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:05.616959 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:05.661984 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:05.710808 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:05.837652 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:05.846532 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240503 02:05:05.858688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.861673 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.871842 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.872103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:05.879911 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.902601 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.958912 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:05.959540 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.960582 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:05.962065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:05.962838 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.004024 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.032575 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.037912 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.041492 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.044106 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:06.056989 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:06.059906 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.060398 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:06.075691 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:06.109839 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:06.213845 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:06.296900 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:06.297731 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:06.301618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.304756 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.305959 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.310663 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.318161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:06.402339 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.404747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.404981 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.422227 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.423388 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:06.434659 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:06.444002 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:06.455389 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:05:06.473256 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:06.484809 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:05:06.485687 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:06.509342 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:06.520572 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.525499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.528014 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.529125 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.530144 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.543836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:06.563824 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:06.594940 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:06.626264 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.628491 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.632272 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.635557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.635998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:06.647356 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.838174 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.840516 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.842676 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:06.846231 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.846479 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:06.862748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:06.879470 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:06.958086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.959101 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.971487 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:06.987560 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:06.989933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:06.990273 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.001477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.006878 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.012698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.014530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.017630 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:07.022003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:07.073493 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:07.092362 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:07.180899 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:07.193406 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:07.198087 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:07.340646 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.340742 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.342149 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:07.345486 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:07.346036 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:07.346477 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.356138 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.373556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.373726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.375432 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.380519 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.385857 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:07.391026 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:05:07.391649 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:07.405485 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.507598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.508422 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.509176 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.510963 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.512137 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:07.519380 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:07.519583 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:07.531958 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:07.546754 29217 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:07.546880 29217 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:05:07.547273 29217 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:05:07.548445 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:07.548754 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:07.548823 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:07.548941 29217 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:07.604159 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:07.659709 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:07.670908 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:07.692279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.693722 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.695513 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.700416 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:07.703919 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.705148 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:07.766026 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:07.826457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.826655 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.837080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.839069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.839167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.842661 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:07.882308 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:07.927636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.932864 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.933970 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:07.935299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:07.954210 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:07.960762 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:07.989671 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:08.030155 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.030580 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.033927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.039956 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.045097 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:08.048394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:08.127836 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:08.158023 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:08.173436 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:08.174639 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:05:08.254261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.254483 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.256075 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.260349 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.261409 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:08.279879 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:08.362104 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:05:08.364511 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.366825 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.379491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:08.386628 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:08.400480 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:08.401032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:08.401854 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:08.407269 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.418401 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:08.420790 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.423039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.428086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.428875 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:08.445567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:08.475795 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:05:08.559542 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:08.618959 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:08.661590 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:08.762362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.762472 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.764430 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:08.764441 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.774406 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.779386 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.783923 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.788276 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.805598 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:08.817855 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:08.819245 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.825474 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:08.837329 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:08.841311 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:08.843477 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:08.922163 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.923708 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:08.924679 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.928432 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:08.929152 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:08.943961 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:08.966630 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:09.033057 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:09.084573 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:09.132861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:09.134562 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.136330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.143036 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.151974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.161551 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:09.230149 29261 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:09.230285 29261 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:09.230787 29261 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:09.232177 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:09.232960 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:09.233024 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:09.233326 29261 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:09.252923 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.253016 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.255659 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:09.257542 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.281740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.282599 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:09.345769 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:09.368149 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.374159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:09.375288 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.376925 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.377513 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.398816 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:09.446683 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:09.448046 29751 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:09.448185 29751 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:09.448587 29751 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:09.449669 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:09.449760 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:09.449847 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:09.450074 29751 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:09.488206 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:09.491061 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:09.491520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.492210 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.497675 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.498817 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:09.509197 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:09.510548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:09.660557 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:09.688989 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.690496 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.691493 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.696946 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.701092 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:09.716672 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:09.732388 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:09.734455 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:09.824503 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.827545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:09.828765 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.840817 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.849570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.858064 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.860304 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.874276 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:09.888070 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:09.895481 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:09.903048 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:09.903406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:09.928463 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:09.944824 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:09.951416 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:09.960826 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:10.012802 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:10.033118 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:05:10.054678 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:10.135913 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:10.169745 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.172338 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.173977 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.174296 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.176753 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:10.193624 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:10.194238 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:10.244699 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.248836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.249634 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:10.251236 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:10.251708 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.255031 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:10.349210 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:10.349407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.349663 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.352254 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.361341 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:10.362005 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:10.365072 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:10.365430 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:10.433471 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:10.439348 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:10.466971 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:10.489959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.493731 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.494344 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.494724 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.495015 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:10.495544 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:10.630761 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:10.657750 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.658703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.665000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.668252 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:10.672425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.677861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:10.764752 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.765316 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.770753 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:10.772014 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.782842 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:10.796350 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:10.804340 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.858253 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:10.859905 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:10.862903 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.864596 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.872359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:10.872769 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:10.885656 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:05:10.899092 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:10.922721 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:10.958622 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:11.071229 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:11.095633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.096963 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.097515 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.098372 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.106998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:11.127424 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:11.129042 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:11.202279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.214656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:11.215782 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:11.217407 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:05:11.222901 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.223912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.248925 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.267401 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.267625 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.279716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.282004 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:11.282384 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.283095 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:11.336560 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:11.374181 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:11.473403 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:05:11.529933 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:11.561007 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.567579 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:11.574149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.579102 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.579970 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.585846 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:11.592015 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240503 02:05:11.628224 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:11.641213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.646301 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.647958 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:11.648561 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:11.655951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.670045 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.746152 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.750627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:11.751523 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.759909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:11.762501 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.776600 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:11.783579 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:11.877393 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.879607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.882913 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:11.892937 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:11.892952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:11.896126 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:11.899926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:11.909226 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.045540 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:12.071713 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.071995 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.076787 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.082974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.083161 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.107131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.203815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:12.206331 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:12.215134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.217732 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.226159 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.229893 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.230376 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.231451 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.237694 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.249630 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.254271 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.274796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.275131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:12.286643 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.325284 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:12.341686 29751 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:12.341843 29751 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:12.342326 29751 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:12.343283 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.343317 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:12.343536 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:12.343590 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:12.343726 29751 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:12.411060 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.413602 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:12.520648 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:12.537252 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.550750 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:12.579557 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:12.581631 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.588620 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.589373 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.596865 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.598449 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.600246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.612111 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.619359 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.621944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.644470 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.655027 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.655661 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.725814 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.727914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.728178 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.730198 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.733214 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.751905 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:12.771327 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:05:12.815814 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:12.861677 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:12.939529 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:12.954650 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.958652 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:12.960819 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:12.960988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:12.969318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:12.979962 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.060961 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.064685 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:13.064860 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.065985 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.068315 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:13.079867 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:13.093070 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:13.106451 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:13.136067 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:13.163070 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:13.177893 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.181428 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.183250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.193254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:13.197386 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:13.203969 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:05:13.204792 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:13.208357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:13.250965 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:13.291635 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.292984 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.294667 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:13.294871 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.296111 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.297299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:13.312965 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:13.486799 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:05:13.506481 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:13.508816 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.510229 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.512764 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.520826 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.549077 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:13.577279 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:13.613703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.619277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:13.623925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:13.625305 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:13.638270 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.639477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:13.643175 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:13.675312 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:13.685912 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.690411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:13.690824 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.692574 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.700264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:13.707070 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:13.799119 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:13.864284 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:05:13.957588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:13.957767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.958750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:13.963420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:13.976058 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:13.999954 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.049808 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.052392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:14.057874 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.059942 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:14.062088 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.065068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:14.075816 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:14.094034 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:14.165671 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.166510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.170611 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.171031 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.179394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:14.190287 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:14.222383 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:14.279861 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:14.287070 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.288451 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.290241 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:14.296568 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.305003 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.310029 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:14.416872 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:14.491685 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.492525 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.494611 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.499800 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:14.502173 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:14.508721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.622952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.623951 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.634125 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:14.634852 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:14.637683 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.638418 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.639822 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:14.641738 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:14.642215 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:14.643538 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:14.647354 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:14.659920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.664212 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:14.670482 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:14.690917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:14.787833 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:14.811496 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:14.995443 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:14.998678 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.000336 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.005306 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.023986 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.025663 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:15.027688 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:15.033943 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:15.034037 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.039927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:15.047611 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:15.055262 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:15.061242 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:15.065358 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.070482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:15.111565 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:15.112341 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:05:15.140712 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:15.172904 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:15.174015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:15.175436 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.175518 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.180518 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.200474 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:15.207412 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:15.251488 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:15.256953 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:15.367928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.369933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.370023 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.374047 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.375006 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:15.378221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:15.392853 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:15.447227 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:15.455745 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:15.465703 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:15.496754 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:15.499827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.510247 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.510545 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:15.523324 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.529423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.608124 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:15.609862 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.611510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.614632 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.619050 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.644810 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:15.648700 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:15.656198 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:15.703002 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.704036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:15.705344 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.706918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.715054 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:15.718849 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:15.722956 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:15.733722 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:15.768411 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:15.886555 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:15.935343 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.935401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:15.939851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:15.940537 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:15.951504 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:15.960978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:15.967948 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:05:16.007607 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:16.070719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:16.071920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.073011 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.083802 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.093712 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:16.095014 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.105296 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:16.106163 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:16.115882 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.127331 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.129402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:16.136407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:16.149637 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:16.256071 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:16.262300 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:16.452767 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.455403 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.461434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:16.462075 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:16.469844 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.488346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.488705 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.491338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.493219 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:16.516392 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:16.525020 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:16.527415 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:05:16.532642 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:16.575228 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:05:16.612272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:16.615383 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:16.616319 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.620044 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.620643 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.620731 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.623998 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:16.640195 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:05:16.791074 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:16.807374 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.807957 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:16.815011 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:16.817142 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.827744 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:16.829178 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.841212 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:16.868963 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:16.875994 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:16.923017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:16.926874 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.933036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.933564 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:16.936043 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:16.939961 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.028923 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.032595 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.034761 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.038244 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.040043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.045332 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:17.132149 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:17.141139 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.145675 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.147881 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.151322 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:17.151422 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.151696 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:17.154232 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:17.345484 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.347256 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.348163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.348912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:17.354312 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:17.361235 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.361842 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:17.388767 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:17.407720 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:17.457170 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:17.470499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.477411 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.477797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.480208 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.495929 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.499087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.500911 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:17.513564 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:17.519603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.524122 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.524163 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.538223 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:17.554226 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:17.573366 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:17.602212 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:17.610509 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:17.700340 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:17.748834 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:17.793522 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:05:17.801211 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:17.852802 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.854468 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.854868 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:17.859355 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:17.872486 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.874979 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.879738 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:17.882576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:17.883927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.884428 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.887827 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:17.888870 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:17.916136 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.007946 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.011241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.014107 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.014158 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.016783 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.019073 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:18.027997 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:18.030900 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:18.037873 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:18.077100 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:18.199694 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:18.215746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.220389 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.225561 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.226204 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.228138 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.234704 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:18.317351 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:18.324977 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.325672 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.326444 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.331336 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.336484 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.337163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:18.380497 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:18.434650 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:18.439258 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.441763 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.446003 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.446534 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.456768 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.462814 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.545045 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:18.545676 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:05:18.546846 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.547626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.548449 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.551265 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.567912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:18.605989 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:05:18.608336 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:18.697467 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:18.749890 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.752599 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.756799 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.757313 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.760037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.772342 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.888142 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.893508 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.896731 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.899091 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.900221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.908155 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:18.912451 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.913017 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:18.924389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:18.927901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:18.935855 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:18.937698 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:18.943282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:19.088896 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:05:19.172008 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:19.206080 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:19.249779 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.256482 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.256893 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.258302 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:19.263908 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:19.266279 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:19.278012 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:19.298139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.302120 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.308463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.310628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.315430 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:19.318748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:19.388664 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:19.413641 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.413748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.416954 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.424901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.432981 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:19.434955 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:19.437371 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:19.527419 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:19.624127 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.625895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.626730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.631280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.641572 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:19.648878 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:19.734723 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.737542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.738171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.739003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:19.740820 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:19.742391 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:19.758074 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:19.776319 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:19.844612 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.844853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.845615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:19.851941 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.860837 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:19.875221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:19.899724 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:19.906049 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:19.955133 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.955871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.956720 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:19.959105 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:19.974555 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:19.981045 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:19.984500 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:20.043512 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:20.046851 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:20.091145 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:20.119012 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:20.162923 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.164458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.167888 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:20.179219 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:20.179878 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:20.185815 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:20.186789 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:20.224346 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:20.275631 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:20.297099 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.297467 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:20.299381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:20.304081 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:20.304371 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.306062 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:20.310488 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:20.316138 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.318033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.324076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:20.327965 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:20.334621 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:20.344372 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:05:20.357810 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:20.362465 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:20.589862 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:20.670421 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.678006 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.680024 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:20.680521 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:20.681989 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.694239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.701303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:20.708163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.710434 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:20.731288 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:05:20.732491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:20.733664 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.736377 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:20.749346 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:20.775501 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:20.780748 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:20.781283 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:20.827713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.830032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.831902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:20.835558 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:20.836634 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:20.843555 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:20.857434 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:20.966266 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:21.028070 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240503 02:05:21.036572 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.038197 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:21.038465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.039316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.039825 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.043887 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:21.050683 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:21.142601 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.146325 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.146595 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.151916 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.155210 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.170054 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:21.252725 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.253065 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.254909 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.259133 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:21.259402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.261173 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:21.269580 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:21.356137 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.356611 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.360713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.362639 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.371181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.379287 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:21.524428 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:21.566784 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.567468 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.568527 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.569768 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.578397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:21.580976 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:21.581023 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:21.663625 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 02:05:21.685540 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.687633 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.698153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.699898 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:21.700876 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:21.701552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.705443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.718472 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:21.721211 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.729641 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:21.737116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:21.746271 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:21.771812 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:21.806802 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:21.841742 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:21.849275 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:05:21.875105 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:21.980279 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:22.067265 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.074586 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.074882 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.074889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.076546 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:22.084204 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.092742 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:22.096875 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.097164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.122181 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:22.131603 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:22.132293 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.132987 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:22.159319 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:22.236703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.239585 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.240303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.250104 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:22.252991 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.263705 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:22.329119 1008 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:22.329247 1008 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:22.329682 1008 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:22.330454 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:22.330637 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:22.330719 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:22.330855 1008 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:22.356511 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:22.379837 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:22.423463 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.424041 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:22.429239 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:22.430713 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.431106 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.440948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:22.453845 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:22.457037 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:22.470328 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:22.515302 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:22.531494 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:22.553423 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.555747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.564229 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:22.564556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.566196 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:22.573714 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:22.592808 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:22.593236 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:22.619812 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:22.660562 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.662325 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.664574 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.665861 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:22.686357 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.690732 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:22.759930 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.760504 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.762631 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.767916 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:22.768282 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:22.770779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.777729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:22.798776 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:05:22.901512 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:22.904443 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:22.924479 31834 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:22.924607 31834 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:05:22.925009 31834 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:05:22.925845 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:22.926043 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:22.926115 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:22.926246 31834 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:22.978938 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.979096 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:22.980823 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:22.981535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:22.998824 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:23.002774 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:23.003983 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:23.028856 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:23.114398 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.116191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.119944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:23.120076 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.120375 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.124130 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:23.138978 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.140194 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.152287 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:23.160894 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:23.173626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:23.178131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.192286 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:23.203059 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:23.223930 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:05:23.297016 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:23.404090 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:23.467981 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:23.477962 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:23.478245 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:23.482631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.487787 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:23.489607 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.497793 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:23.509841 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.538416 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.546161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.554895 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:23.566442 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.571033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.572136 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:23.579069 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:23.609328 32327 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:23.609475 32327 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:23.609902 32327 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:23.610978 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:23.611083 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:23.611160 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:23.611290 32327 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:23.657574 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.659240 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.661062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.663044 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:23.669054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.673485 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:23.676000 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:05:23.838706 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.841048 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.841249 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.843537 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.854279 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:23.857609 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:23.913990 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:05:23.930044 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:23.972074 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:23.973852 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.977504 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:23.978004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:23.981117 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.003294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.010061 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.093895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.094134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.099570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.107195 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.107719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.113880 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:24.128656 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:24.185590 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.186486 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.187022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.195363 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.200754 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.208108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:24.221174 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:24.273371 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:24.416543 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.418644 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.419493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.433295 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.441570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.444870 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:24.452725 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:24.456352 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:24.494967 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:24.548905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.550626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.551478 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.556396 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.560714 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.562083 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.564805 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.579249 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.587939 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.595919 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.606194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.614265 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:24.729174 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:24.780771 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:24.780937 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:24.894574 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:24.908527 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.913962 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.916100 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.917544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:24.919824 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:24.928995 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:24.935945 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.961464 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.961755 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:24.967089 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.968713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:24.970312 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:24.977102 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:24.980913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:24.987243 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:24.990207 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:25.001495 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:25.021903 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:25.073915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.074738 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.078235 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.086922 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:25.087612 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:25.093940 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:25.115653 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:25.137868 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:25.182058 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:05:25.286710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.287631 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.292867 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:25.298774 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:25.300429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.308542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:25.390345 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:25.407594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.408253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.412598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.415905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:25.434859 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:05:25.435701 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:25.438274 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:25.440667 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:25.445883 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:25.474437 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:25.524035 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.524159 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.524255 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:25.525015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.528262 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.536746 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:25.575475 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:25.626730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.627988 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.640810 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.641039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:25.641060 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.641533 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:25.718585 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:25.806758 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:25.846474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.847590 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.849782 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:25.851936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.868335 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:25.868772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:25.880785 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:25.966063 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.969656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:25.970142 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:25.978729 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:25.987493 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:25.996785 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:25.999706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.002180 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.007834 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.016345 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:26.029156 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240503 02:05:26.030931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.031474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.033186 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:26.139824 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:26.225951 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:26.235706 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:26.358474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:26.359484 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:26.359547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.361202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.362357 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.375145 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.383563 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:26.389669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.419236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.426144 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.428565 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.429381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.436738 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:26.531209 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.535604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.535889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.539937 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.548264 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.550652 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:26.616556 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:26.626693 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:26.694093 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.696517 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.698267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.699693 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.702035 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:26.705282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:26.747418 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:26.780277 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 02:05:26.844039 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.845777 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.850512 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.853113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.860424 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:26.862936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:26.921108 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:05:26.922143 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:26.935174 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:26.944468 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:26.949566 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.953891 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:26.956938 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:26.962854 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:26.978531 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:26.987952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:27.024070 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:27.039649 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.041772 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.042260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.043097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.047173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:27.059358 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:27.125499 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:27.163235 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:27.177990 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:27.269642 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:27.282671 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.284127 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.287257 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.292025 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:27.299870 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.300045 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:27.331132 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:27.346652 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:27.406407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.409359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.416430 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.417801 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:27.424482 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:27.425863 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:27.427094 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:27.441022 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:27.443460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.445745 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:27.446671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.463857 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:27.478096 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:27.484537 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:27.493899 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:27.637727 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:27.671980 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:27.672272 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:05:27.736189 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:27.778945 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.784031 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.794368 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.795202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:27.800096 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:27.803243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:27.838596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:27.845067 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.845521 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.846879 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.856911 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.857719 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:27.865885 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:27.915802 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:27.956184 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.956967 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.957950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:27.962894 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:27.963176 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:27.972254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:27.986656 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:28.011394 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:28.144003 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:28.149751 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:28.164860 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.166841 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.167275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.170593 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.179450 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.184124 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:28.270339 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:28.279495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.282120 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.283248 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.283915 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.290799 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:28.293844 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:28.336217 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:28.385857 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.388901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.392135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.394169 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:28.394340 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:28.395910 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.404430 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:28.413518 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:28.506233 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.506546 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.510840 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.514053 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.525301 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.533804 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:28.633818 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:28.707726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.711056 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.713069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.713919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.720999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.740880 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:28.757853 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:05:28.758553 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:05:28.759409 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:28.839237 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:28.846760 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.847429 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.848070 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.848407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.865671 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.867568 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.870183 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:28.881284 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:28.881646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:28.883671 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:28.891181 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:28.912529 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:28.929235 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:05:29.094514 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:29.229604 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.230222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:29.236002 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:29.238399 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.239703 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.240168 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.249601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:29.249882 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:29.257295 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:29.270103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.279552 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.284240 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.285226 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:29.325469 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:29.363740 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:29.379830 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.385562 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.385819 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:29.388362 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.397231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.398866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:29.439520 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:29.514936 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:29.568209 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:29.584388 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.584810 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:29.585314 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.585777 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.586907 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.588690 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:29.608958 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:29.657555 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:29.682123 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:29.701442 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.702898 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.704981 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.705788 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.706815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:29.714941 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:29.740526 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:29.811486 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.811823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.822746 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:29.825237 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:29.826267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.838661 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:29.855486 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:29.909670 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:29.913899 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:29.923233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.925482 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.929191 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:29.931314 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:29.940265 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:29.941114 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:29.976606 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:29.987305 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:30.011031 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:30.039520 2792 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:30.039650 2792 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:30.040114 2792 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:30.040815 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:30.041055 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:30.041131 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:30.041251 2792 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:30.084386 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:30.132692 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.136971 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:30.138482 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.142494 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:30.143460 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.150214 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:30.238181 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:05:30.258813 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.265775 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.274406 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:30.274933 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:30.277667 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.282374 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.282905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.296743 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:30.301602 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:30.304023 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:30.317273 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:30.323025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:30.339504 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:30.339843 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:30.364694 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:30.465066 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:30.466503 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:30.629694 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:05:30.632258 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:30.637234 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.647627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:30.653808 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:30.658445 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.666798 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:30.667001 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.669807 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:30.681591 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.683312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:30.688450 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:30.692273 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.701821 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:30.707369 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:30.798462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.807476 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:30.807880 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:30.809377 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.812352 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:30.828130 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:30.866230 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:30.873942 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:30.941903 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:31.010604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.016551 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.019142 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:31.019696 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.020718 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.027545 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.122257 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.126008 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.126710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.134773 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:31.135721 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:05:31.136447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.143891 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:31.193734 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:31.232460 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.238569 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.240514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.248340 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.257732 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:31.258350 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.340018 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.342478 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.344038 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.351948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.353590 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.356621 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:31.392306 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:31.433239 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:31.495787 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:31.557516 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.559191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.559821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.560010 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.568537 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.568948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:31.676677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.680919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.688477 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.695127 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.696681 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.696846 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:31.698323 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.716915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:31.719208 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:31.744511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.747939 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:31.749675 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:31.781356 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:31.855580 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:05:31.856282 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:31.888660 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:31.991014 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:05:31.996471 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:32.030222 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:32.038658 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.040426 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.041409 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.051944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.052091 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:32.059675 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:32.091884 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:32.097827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.107587 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.115036 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.121404 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.122607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.137874 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:32.181542 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:32.220227 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.220892 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.224329 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.224586 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.226624 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.226964 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:32.231341 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:32.240168 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:32.260800 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:32.273713 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:32.374688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.381953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.384495 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.389283 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:32.390504 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.393319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:32.408622 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:32.477676 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:32.487042 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:32.520172 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:32.533046 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.533483 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:32.538023 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:32.538887 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.539908 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.540395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.543915 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:32.554478 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:32.570217 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:32.644145 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.644599 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.645231 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.646205 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.647600 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.649457 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:32.658389 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:05:32.733325 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.734975 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.741834 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.749172 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.752758 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.758373 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:32.903266 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:32.960151 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.964082 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:32.966856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:32.968243 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:32.979996 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:32.996006 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:32.996351 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:32.996767 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:33.010464 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:33.093611 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.094331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.103350 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.114813 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.115106 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:33.116724 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:33.120029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.120270 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.120268 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:33.133304 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:33.133486 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.140498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.174561 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:33.343065 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:33.344885 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:33.431507 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:33.465000 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:33.470729 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.476128 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.477517 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:33.482272 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.503412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:33.507839 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:33.511631 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.523010 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.527807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:33.533370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.534631 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:33.556000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.557512 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:33.632486 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:33.651018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:33.652186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.653254 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.654124 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.667258 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:33.669891 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:33.766906 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:05:33.790606 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:05:33.838158 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.839526 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:33.843851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:33.843926 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.853734 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.853734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:33.965682 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:05:33.966262 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.966575 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:33.968453 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:33.974448 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:33.976747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:33.979399 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:33.987192 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:33.993432 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:34.078033 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.080646 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.082850 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.091958 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:34.106602 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:34.107234 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:34.116366 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:34.132560 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:34.175123 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:34.175125 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.176390 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.176997 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:34.177781 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.193593 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:34.401515 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:34.412403 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.414242 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.427182 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:34.429849 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:34.430589 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.431923 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.431955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:34.470950 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:34.472287 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:34.493584 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:34.539572 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.542523 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.563438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.565551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.568555 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.568948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.569320 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:34.574551 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:34.582160 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:34.585884 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:34.591055 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.593299 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:34.611423 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:34.667357 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:34.871805 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:34.875819 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:34.903142 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:34.921015 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.924252 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.929097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:34.930335 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:34.936394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:34.938706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.949852 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.950654 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:34.955049 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:34.963711 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:34.968966 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:34.979269 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:34.979811 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:34.981870 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:34.986912 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:35.010483 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:35.060796 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:35.084081 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.091156 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.091696 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.094015 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.096088 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.115531 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:35.238415 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:35.288056 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.289983 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.290625 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.293480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.296289 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:35.302784 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.397745 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.403410 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.403486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.418042 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:35.419593 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.431717 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:35.447993 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:35.453640 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:35.477741 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:05:35.478694 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:35.491947 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:35.522039 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.523136 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.524634 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.527709 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.541085 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.556442 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:35.624181 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.627060 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.628935 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.630993 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:35.639087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.640096 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:35.741925 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:35.845167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.847954 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.849488 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.849702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:35.852406 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.879879 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:35.900717 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:35.912719 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:35.933048 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:35.959061 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.963372 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:35.964581 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:35.967499 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:35.970322 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:35.970628 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:35.989393 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:36.008827 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:36.033980 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.033967 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.035449 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.035707 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.039093 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:36.050010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:36.144834 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:36.238768 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:05:36.275154 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240503 02:05:36.286808 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.289009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.294855 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.301568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.304101 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:36.318346 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.394659 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:36.397589 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:36.399590 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:36.400820 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.407485 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.411938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.414063 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.414934 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:36.488207 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:36.513746 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.513746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.516197 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:36.523521 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:36.532315 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.535168 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:36.537333 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.616763 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.617514 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.619896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.622596 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.622646 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.632705 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:36.786221 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:36.841746 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.843271 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.844506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.844879 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.853760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.862825 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:36.899513 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 02:05:36.923794 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:36.935029 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:36.954623 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.956588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.963925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:36.971346 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:36.979658 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:36.985611 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:36.996038 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.007151 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:37.010185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.010354 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:37.030613 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:37.032871 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:05:37.032910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:37.033442 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:37.034281 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:37.078461 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:37.148190 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:37.209547 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:37.326881 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.331593 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.335604 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:37.345012 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:37.347946 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:37.354491 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.355578 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:37.364019 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:37.367983 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:37.375602 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:37.383337 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.384263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.393366 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:37.395908 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.401715 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:37.411075 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.439252 5587 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:37.439421 5587 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:37.441403 5587 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:37.442241 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:37.444276 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:37.444353 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:37.444504 5587 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:37.456470 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:37.506857 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.507222 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.509256 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.511087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:37.526626 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.532821 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:37.548130 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:37.693442 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:05:37.705471 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.707531 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.710738 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:37.710898 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.720088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.733199 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:37.749114 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:37.820619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.822103 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.831112 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.841873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.842761 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:37.851235 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:37.856822 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:37.921945 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:37.937486 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.940346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:37.941210 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:37.941251 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:37.945005 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:37.950063 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:38.009411 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:38.025615 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:38.042500 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.042786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:38.044576 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.054459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.057799 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:38.063500 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:38.106854 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:38.170754 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:05:38.259644 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:38.259744 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.262620 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.265687 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:38.274443 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.279388 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:38.302786 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:38.377869 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.379498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.384169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.394686 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.398115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:38.400589 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:38.402015 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.410343 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:38.412048 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:38.423141 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:38.425269 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:38.428186 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:38.432899 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:38.454344 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:38.470029 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:38.547964 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:38.661429 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:38.686192 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:38.761085 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.764753 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:38.766095 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.767488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:38.772238 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:38.776079 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.790128 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:38.798225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.799015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:38.803031 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:38.806243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:38.822798 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.824998 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:38.844483 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:38.862483 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:05:38.927927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.929585 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.931457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:38.933555 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:38.935932 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:38.952243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:38.983170 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:39.141420 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:39.141861 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:39.143683 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.144958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.145807 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:39.147497 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.167444 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.248548 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.249081 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.249743 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.252723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.253520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:39.274464 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:39.274924 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:39.346359 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:05:39.347370 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:39.350417 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.352396 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.353215 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.356348 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.362361 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.372265 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:39.415938 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:05:39.459635 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:39.466081 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:39.467005 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.467344 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.472784 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.482164 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.482448 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:39.485473 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:39.503233 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:39.555842 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:39.650117 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:39.663018 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.664608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.666627 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.668285 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.684489 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.687048 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:39.779738 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.782152 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:39.786267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.788486 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.797657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.799549 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:39.848407 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.853443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.858413 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:39.859445 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:39.859871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:39.873225 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:39.887956 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:39.900918 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:05:39.929409 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:39.933701 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:39.974946 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:39.978673 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:40.121042 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.122382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.128577 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.142647 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.145707 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:40.146173 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:40.149170 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:40.163342 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:40.207631 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.209455 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.215948 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.219058 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.220347 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:40.229189 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.289351 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.290588 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.296452 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.303205 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.311975 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.320508 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:40.417212 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:40.470180 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:05:40.490689 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:05:40.520670 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:40.521762 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.525964 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.527479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:40.531095 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.540935 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.549810 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:40.606392 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:40.626186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.627579 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.629293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.630309 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.632359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.651530 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:40.652333 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:40.729286 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.730691 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.731269 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.742873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.746212 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.749923 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:40.762259 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:40.848603 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:40.851195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:40.852679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:40.853597 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:40.859915 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:40.866330 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:40.867298 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:41.001801 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:41.018018 6695 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:41.018148 6695 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:41.018575 6695 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:41.019799 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:41.019891 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:41.019953 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:41.020069 6695 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:41.035358 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.039700 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:41.040858 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:05:41.042054 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.042834 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.052335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.068908 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:41.075245 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:41.118366 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:41.160857 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.169749 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.173297 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.173398 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:41.174840 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:41.184239 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:41.190968 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:41.210687 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.216778 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.219002 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.223399 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:41.223708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.223889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.257179 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.260151 31834 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:41.260262 31834 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:05:41.260633 31834 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:05:41.262624 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:41.262738 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:41.262777 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:41.262912 31834 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:41.397178 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:41.415287 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:05:41.473739 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:41.552983 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.557459 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.560958 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.563441 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.569635 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.572048 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.580245 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.587405 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:41.600052 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.608664 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.611007 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:41.615725 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:41.624660 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:41.651065 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:41.714403 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:41.714609 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.714938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.715867 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.718866 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.723228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:41.742161 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:41.849648 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:41.855160 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:41.905149 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.906184 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.908139 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:41.909124 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:05:41.910347 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:41.927254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:41.928469 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:42.024724 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.025696 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.027669 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.029641 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.030329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.032714 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:42.040052 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:42.115164 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.133054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.133265 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.133867 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.136444 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240503 02:05:42.136919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.145251 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.154496 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:42.155108 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:42.155615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:42.225827 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.243000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.244215 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.248011 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.248229 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.249785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:42.265496 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:42.319171 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:42.322176 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:42.379560 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:42.451085 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:42.452559 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.453003 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.454161 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.461056 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:42.462760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.469228 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:42.477674 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:42.566233 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.570550 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.585732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.585915 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.588793 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:42.593777 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:42.613293 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:42.616041 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:42.616439 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:42.620360 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.633327 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.633965 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:42.641041 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.643503 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:42.643903 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:42.946179 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:42.951087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.951859 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:42.952123 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:42.956954 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:42.966270 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:42.983804 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:42.994357 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.003914 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.004124 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.020540 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.023939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.026484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:43.037849 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:43.047952 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:05:43.077854 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:43.084705 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:43.126466 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.127185 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.127933 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.130235 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.130527 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.131220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:43.180374 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:05:43.240314 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:43.248657 7331 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:43.248800 7331 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:43.249224 7331 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:43.250121 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:43.250209 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:43.250270 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:43.250404 7331 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:43.254285 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:05:43.296945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:43.298889 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.298978 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.305815 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.313665 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.320394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.446451 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.448016 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.453308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.460642 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.469998 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.469995 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:43.532294 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:43.537307 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:43.543623 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:43.549075 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.556048 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.557214 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.557440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.567337 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.578063 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:43.653132 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.654105 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.654938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.657506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:43.660686 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.669814 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:43.709421 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:43.771224 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:43.872179 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:43.880416 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.883530 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:43.885021 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:43.885442 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:43.886940 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:43.890578 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:43.998828 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:44.010890 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.011159 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.014561 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.019215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.020140 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:44.025713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.027271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.029047 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.029439 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.031992 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:44.051209 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:44.053092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:44.070165 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:05:44.085559 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:44.200359 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:44.217851 7876 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:44.217974 7876 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:44.218395 7876 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:44.219127 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:44.219352 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:44.219448 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:44.219584 7876 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:44.258476 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:44.337801 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:44.368955 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.380594 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.392894 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.394976 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:44.403954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.405743 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.406263 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.406770 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.414530 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.444950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.457607 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.458642 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:44.536851 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.540647 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.541496 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.544723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.548036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:44.553097 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 02:05:44.553678 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:44.564862 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:44.566953 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:44.594489 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:44.597205 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:44.648308 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:44.682233 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:44.685133 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:44.746924 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:44.748750 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.752120 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.753084 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.753311 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.753332 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.755342 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:44.830252 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:44.855705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.856020 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.860527 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:44.866922 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.869338 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.872828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.956801 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.958415 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:44.960201 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:44.969571 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:44.971170 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:44.971398 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.031479 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:45.069481 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.069887 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:45.074748 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.074781 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:45.078845 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.082918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.087994 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:45.096076 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:45.117848 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:45.181545 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:45.283644 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.287334 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.293475 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.293931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:45.298225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.307601 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.400141 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.411370 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:45.432960 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.434787 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.439514 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.439924 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.450901 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:45.452358 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:45.461169 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:45.463348 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.467435 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.473162 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.477936 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.485289 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:45.501199 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.509035 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:45.534256 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:45.582875 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:45.695998 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:05:45.742626 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:05:45.811206 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.818248 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.818277 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.820250 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.823590 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.824273 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.826833 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.832572 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.832594 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.832684 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:45.842228 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:45.860667 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:45.861212 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:45.949844 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:45.951025 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:45.962016 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.964421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:45.966140 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:45.966864 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:45.975917 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:45.976611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.162952 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.164927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.166296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:46.170809 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:46.181849 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:46.184988 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.188993 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.273792 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.278916 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:46.279378 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.279628 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.281710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.282459 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:46.349095 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:46.375980 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.377475 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.381166 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.382378 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:46.390304 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:46.395897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.399894 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:46.487587 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:46.487690 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.488428 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.494271 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.497687 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:46.504087 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:46.513995 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:05:46.514596 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:46.548936 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:46.696178 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.700914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.701154 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:46.703542 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.705426 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.732064 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:46.741485 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:46.749831 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:46.802444 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:46.805106 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.807121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.809525 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:46.816783 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:46.819370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.831775 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:46.840097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:46.855681 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:46.887343 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:46.887665 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.888283 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.896989 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:46.900141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:46.904388 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:46.966430 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:05:46.993083 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:47.032596 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:47.040701 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:47.096709 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:47.126641 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.128686 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.130646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:47.135054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:47.143904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.147915 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:47.183724 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:47.196787 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:47.228617 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.229396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:47.233007 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.240080 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:47.241324 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:47.242312 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.248106 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:47.336233 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.336880 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.337762 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:47.348187 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.348708 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:47.349747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:47.350515 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 02:05:47.449083 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.451439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:47.452883 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.457373 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:47.458232 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:47.474741 8536 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:47.474874 8536 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:05:47.475241 8536 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:05:47.477581 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:47.478116 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:47.478183 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:47.478287 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.478421 8536 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:05:47.481278 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:47.523458 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:47.581877 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:47.612926 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:47.642776 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.643527 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:47.647713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:47.648149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.655134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.673636 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:47.677248 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:47.765050 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:47.772068 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.779096 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:47.784986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:47.785342 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:47.789018 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:47.795069 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.809917 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.810302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.817266 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:47.830452 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:47.840649 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:47.844343 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:47.846804 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:47.847613 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:47.855825 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:47.883208 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:05:47.885946 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:05:48.131196 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.133790 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.134703 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.155268 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:48.157498 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:48.169340 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:48.175992 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:48.197448 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.201402 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.203171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:48.203346 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.206415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.207234 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:48.294325 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:48.298091 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.300314 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.300843 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.302151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.304232 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:48.328796 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:48.402186 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:48.499948 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:48.511718 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.512411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.514178 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.521745 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.528412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:48.544368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:48.616780 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.619323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.623014 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.625567 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:48.631232 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.636654 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:48.705463 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:48.717849 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:48.721062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.722725 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.723521 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.726361 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:48.742264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:48.750933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:48.751283 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:48.841519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.843192 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.843539 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:48.846544 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:48.853437 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:48.859691 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:48.967851 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:48.981145 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:49.034811 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.037622 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.038237 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.038479 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.038743 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.043113 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:49.135462 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:49.175797 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.179107 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:49.180658 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:49.185045 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:49.186743 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.186815 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.188076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.190372 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.196170 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:49.199628 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:49.204300 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.228627 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.236267 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.252290 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:49.354686 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:49.359035 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:49.396641 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:05:49.418792 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:49.424612 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:49.532917 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:49.535689 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.536600 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.545629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.553923 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.558331 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:49.559500 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.578145 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.579308 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.583202 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:49.583452 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 02:05:49.584484 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:49.588516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.589176 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:49.594465 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:49.598237 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.705005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.708822 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.710022 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.712848 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.719182 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:49.721581 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:49.738770 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:49.783393 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:49.808372 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:49.908131 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:49.911204 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:49.914191 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:49.914628 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:49.914988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:49.918927 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:49.921960 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:49.928373 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:49.973491 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:50.029353 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.034269 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.039948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:50.040848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.044384 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.049304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:50.119897 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:50.134955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.136381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.137229 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.139007 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.139313 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.141728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:50.152920 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:50.241580 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.241683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.243072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.244439 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.244444 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.273645 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:50.275935 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:50.295604 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:50.296450 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:50.316262 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:05:50.347352 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:50.414894 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:50.455857 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.456955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.458076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.461068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.461323 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:50.465003 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.577039 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.583747 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.585093 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.588071 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.589066 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.589643 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.592941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.614960 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:50.622469 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.626506 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:50.627941 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:50.636281 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:50.640419 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:50.777770 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:50.946704 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.951853 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.968174 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.971201 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:50.974808 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.977799 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:50.982074 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:50.988554 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:50.989517 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.014696 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:51.017246 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.029529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.121932 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.122967 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.124267 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:51.132948 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.132979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.144127 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:51.153493 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:05:51.158962 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:51.227447 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:51.256991 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:51.292183 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:51.315254 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.316129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.320147 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.322690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.327963 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.330332 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:51.414530 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:51.442250 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.442440 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.446172 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:51.447619 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.447888 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.459100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:51.540930 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240503 02:05:51.549363 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.549924 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.552207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.555615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.560266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.570760 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:51.610102 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:51.656464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.656627 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.660385 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.660957 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.661087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.667318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:51.705653 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:51.731215 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:51.746299 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:51.813629 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:51.825062 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:51.845535 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:51.870354 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:51.873672 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:51.874092 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.874091 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:51.884928 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:51.894291 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:51.952215 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:51.987103 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 02:05:52.008181 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.013806 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.019232 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.021806 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.023068 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.037226 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:52.038673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:52.044051 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.046721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.047210 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.057646 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:52.070994 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:52.090468 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:52.090838 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:52.124606 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:52.152678 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:52.205540 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:52.281591 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:52.289160 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:52.371487 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.380484 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.381666 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.384922 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.392270 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.424186 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:52.432345 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.436240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.437954 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:52.438900 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.457756 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:52.458101 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:52.468009 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:05:52.468634 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:52.475373 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.562144 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.563127 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.568179 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.571468 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:52.579347 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:52.580401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.594178 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:52.644135 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:52.675034 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:52.698637 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:52.710819 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:52.752187 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.752779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.757902 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.762390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.762665 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.767735 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:52.799851 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:52.817299 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:05:52.878513 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:52.881966 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:52.882527 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.884889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.888370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:52.892838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.904330 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:52.990738 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:52.991703 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.995024 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:52.996760 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.011807 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.011905 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:53.074551 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:53.098912 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.099113 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.099678 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.101663 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.105798 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.130584 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.311022 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.312834 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.314802 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.316994 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.319293 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.329669 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.349301 10237 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:05:53.349434 10237 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:05:53.349880 10237 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:05:53.351044 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:53.351290 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:53.351363 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:05:53.351485 10237 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:53.433794 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.438838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.446101 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:53.448009 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:53.449995 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.457618 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.471096 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.471812 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.472229 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:53.482457 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:53.484606 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.485663 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.498440 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.513499 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:53.578433 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:53.608448 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:05:53.679636 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:53.827872 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.829905 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.839248 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.842859 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.844252 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.844969 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:53.860239 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.864136 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.881709 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.881980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.882319 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:53.892647 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.980847 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.982668 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:53.985587 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:53.991300 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:53.991964 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:53.993731 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:54.005950 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:54.006383 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:54.113664 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:54.153326 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:54.187472 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.189337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.198846 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.203943 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:54.204456 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.217069 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:54.283123 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:54.302700 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.303418 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:54.304260 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.310428 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.318769 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:54.324370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:54.343565 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:54.410100 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:54.413933 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:54.414361 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.420233 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:54.422103 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:54.430380 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.437151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.437312 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:54.505379 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:54.513841 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:54.522081 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.524281 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.525645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.526924 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.539086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:54.549796 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:54.551887 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:54.568926 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:54.584940 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:54.595649 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:05:54.657742 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:54.721225 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:05:54.735978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.736297 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.738401 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:54.741343 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.748186 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.759435 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:54.768127 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:54.858934 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:54.865897 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.867800 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.869776 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.870159 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:54.879125 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.884114 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:54.905709 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.916266 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:54.922755 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:54.930863 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:54.938912 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:54.941094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:54.948472 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:54.978984 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:54.979558 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:55.001771 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:55.014075 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:55.087797 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:55.091778 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:55.100018 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:05:55.193689 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.198421 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.201189 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.202425 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:55.210587 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:55.211022 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:55.227612 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.288352 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.290056 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.294752 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.296463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.298826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.320467 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:55.354698 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:05:55.373723 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.374073 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.374122 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.374512 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.391996 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.392803 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:55.483565 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:55.616392 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.619256 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.619585 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:55.621699 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:55.623104 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.623921 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:55.636683 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.714599 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.720772 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:55.723044 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.730346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.732861 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.733660 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.829248 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.829855 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.840694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.850389 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:55.852823 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.853762 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.946331 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:55.950412 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.952188 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.952708 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:55.971400 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:55.974759 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:55.989125 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:56.107595 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:56.148156 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.151585 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.153936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.155694 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.155854 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:56.175173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:56.186836 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:56.219924 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:56.229487 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:56.281076 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.285934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:56.286957 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.287081 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:56.288949 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:56.290396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.290920 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.304502 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:56.319926 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.323309 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.337296 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.342696 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:56.457275 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:56.574059 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:05:56.622558 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:56.645357 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:56.645586 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.653252 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:56.657244 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:56.658427 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:56.661628 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:56.669056 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.683691 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.686829 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.690872 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:56.693164 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.694095 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.702360 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:56.715387 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:56.721007 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:56.728848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:56.773221 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:05:56.794939 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.800956 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:56.807235 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:56.809466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:56.816977 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:56.823906 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:56.997799 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:57.005319 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 02:05:57.025658 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.026293 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.029053 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.029856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.041955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:57.050487 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:57.051163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:57.075918 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:57.099948 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:57.103915 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:57.118333 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:57.123955 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:57.131626 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.134582 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.135004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.137701 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:57.148617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.155983 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:57.161099 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:57.250906 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.251179 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.251801 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.255941 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.256202 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:57.271710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.348166 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.351305 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.352180 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.353549 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.354336 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.361853 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:57.440239 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:57.445923 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:05:57.448594 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:57.503649 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:57.524753 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:05:57.563227 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:57.565260 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:05:57.567371 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.567518 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.570196 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.576427 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.584694 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:57.601853 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:57.602528 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:57.624555 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:57.636654 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:57.676895 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.681074 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.682682 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.697314 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.701654 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:57.703884 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.705933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.726872 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:57.729868 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:57.737561 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:57.749229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:57.763423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:57.765283 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:57.778528 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:58.035162 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.038173 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.040294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.051961 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:58.053491 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.059405 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.084303 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.103499 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.115311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.118839 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.127944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.133939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:58.169360 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:58.187979 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:58.219875 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.224144 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.228376 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.232272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.236783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.247371 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:58.414700 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.415436 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.416316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.418093 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.433185 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:58.437901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:58.474814 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:05:58.536140 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.539443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.539958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:58.540714 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.544903 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.569157 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:58.578442 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:58.645051 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.645557 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:58.645977 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:58.652076 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.652513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:58.660382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:05:58.662869 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:58.667265 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.748343 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.748929 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.750303 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.751080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.752935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.760052 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:58.797235 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:05:58.868366 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:58.962458 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.964239 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.967061 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:58.968122 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:58.973989 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:58.988055 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:59.001448 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:05:59.084745 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.087963 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.090660 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.100837 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.102185 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:59.103982 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:59.120126 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:59.132642 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.137549 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.138420 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.148030 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.148852 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:59.168690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:59.177901 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:59.182658 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:05:59.222926 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:59.292563 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:59.446905 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:59.461850 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:59.465245 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:05:59.466121 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:05:59.466431 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.468816 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.470186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.478210 12090 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:05:59.478358 12090 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:05:59.478950 12090 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:05:59.480353 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:05:59.480506 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:05:59.480559 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:05:59.480690 12090 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:05:59.485910 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.531495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.533560 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.535260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.537012 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.542969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.559026 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:59.568961 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:59.618711 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:05:59.619576 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:05:59.634603 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:05:59.641453 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:59.642131 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.644228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.645592 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.646106 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.646209 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:05:59.654400 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:05:59.660295 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:59.661883 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:59.787590 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:05:59.790879 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:05:59.791705 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:05:59.798965 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.805420 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.809650 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.819680 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.820884 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:05:59.827114 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:05:59.934589 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:05:59.955621 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.961917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:05:59.963001 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:05:59.964049 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.965004 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:05:59.966970 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:00.046996 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:00.064937 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.066244 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.074479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:00.075114 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:00.096045 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:00.096470 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.100383 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:00.103220 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:06:00.138983 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:06:00.158900 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.159924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.161710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.163343 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:00.169422 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:00.171957 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:00.176743 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:00.247314 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:00.350847 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:06:00.416096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.416499 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.417323 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.435143 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:00.437832 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:00.452019 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.519837 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.520637 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.525207 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:00.526271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.530315 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.549880 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:00.550155 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:00.581307 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.589741 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.591349 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.591615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:00.601037 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.613772 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:00.657341 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:06:00.794049 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:00.897642 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.898165 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.900650 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.902123 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.931914 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:00.932490 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:00.933441 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:00.959604 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.962620 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:00.963469 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:00.972033 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:00.982851 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:00.987526 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:00.991148 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:01.068720 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.073022 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.084405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.087139 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.089421 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:01.098134 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:01.124238 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:01.135802 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:01.265908 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.266198 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.276260 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.276324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.279134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:01.291289 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:01.296217 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:01.392269 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.392422 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.399971 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:01.406937 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.407326 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.416975 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:01.428907 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:06:01.505563 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.506546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.514292 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:01.514878 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:01.515012 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.515739 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.603359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:01.605151 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:01.605492 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.607169 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.611048 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.612370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:01.617640 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:01.677156 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:06:01.725788 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:01.748317 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:01.819607 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:01.824539 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.827721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.830050 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:01.830539 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.835897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:01.848243 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:01.848922 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:01.881995 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:01.926781 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:01.953855 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:01.954265 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:01.954497 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:06:01.958065 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.964825 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.965423 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:01.965885 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:01.968845 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:01.969571 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:01.973495 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:01.975718 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:02.000892 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:02.011775 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.016307 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:02.018440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.030437 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:02.036948 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:06:02.060132 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:02.157420 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:02.249949 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:02.326359 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:02.338294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.338759 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.341131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.345722 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.356276 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:02.359175 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:02.361207 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.364154 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.367102 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.367527 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.373066 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.389644 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.390231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:02.420989 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:02.451984 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:02.499913 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.503348 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.504832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.507323 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:02.515498 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.517732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:02.522040 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:02.643481 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:02.648582 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:02.698065 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.701193 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.702353 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.703390 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:02.703567 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.731091 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:02.733578 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 02:06:02.740814 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:02.816022 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.816658 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:02.818178 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.818396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.820392 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.822124 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.869177 12192 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:02.869323 12192 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:02.869709 12192 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:06:02.870780 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:02.870935 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:02.870988 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:02.871125 12192 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:02.919212 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.919683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:02.920893 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.931667 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:02.932960 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:02.942644 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:02.962846 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:02.964797 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:03.024932 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.024968 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.027132 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.029876 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:03.050850 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:03.055276 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:03.069095 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:06:03.163252 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:03.245631 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.246976 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.254359 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:03.256265 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.262111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.275337 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:03.350661 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:03.359488 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.359959 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.365032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:03.366585 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:03.367789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.390086 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:03.400943 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.409160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.410264 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.413489 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.418007 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:03.428710 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:03.431479 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:03.458195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:03.642377 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:03.659619 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:03.753887 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.755585 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.760864 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.762463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.763478 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:03.781771 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.791271 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:03.797097 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.826013 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.826730 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:03.827607 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:03.837882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:03.883565 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:03.928836 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.930590 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:03.931136 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.935452 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:03.937139 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:03.955893 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:03.962335 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:04.139905 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.140899 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.143574 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:04.146692 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.147257 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:04.168507 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:04.185622 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:04.252238 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:04.256430 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.258000 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:04.258569 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.259528 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:04.267144 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:04.267283 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.279392 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:04.293989 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:04.367713 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:04.367786 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:04.368808 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:04.371506 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.374945 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.380815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:04.382583 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:04.384001 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:04.388689 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.388887 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:04.406019 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:04.470423 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:04.484803 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:04.486114 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.486582 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.488603 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.495888 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.503899 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:04.509047 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:04.634498 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:06:04.658934 8536 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:04.659073 8536 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:06:04.659593 8536 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:06:04.660797 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:04.660897 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:04.660996 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:04.661267 8536 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:04.690728 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:04.694165 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:06:04.696097 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.696753 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.698832 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:04.700510 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:04.701840 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.704919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:04.750769 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:04.825392 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:04.826798 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.833429 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.835830 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.836176 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.845515 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:04.849758 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:04.850436 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:04.875932 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.878576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:04.879802 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:04.885773 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:04.905360 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:04.905813 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:04.958107 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:05.003228 12843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:06:05.003985 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:05.031672 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:05.067404 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:05.201663 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:05.208289 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:05.210927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.211217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.213390 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:05.234170 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:05.237435 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.239990 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.246806 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.252774 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.256263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.267771 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:05.280797 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:05.340212 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:05.362229 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.367017 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.369956 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.377228 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:05.389710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.401909 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:05.463333 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:05.545729 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:05.577428 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.577493 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.577649 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.585285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.586638 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:05.594161 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:05.611460 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:05.680742 14487 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:05.680891 14487 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:05.681344 14487 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:06:05.682425 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:05.682523 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:05.682610 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:05.682739 14487 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:05.693308 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.694412 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:05.694481 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:05.698586 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.700855 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.704711 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:05.706251 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:05.796051 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.796569 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.798097 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.808444 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.809521 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:05.819501 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:05.854387 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:05.903956 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:05.905054 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.914391 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:05.914942 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.916710 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:05.928845 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:06.009968 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:06.087098 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:06.106040 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:06.124404 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.124840 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.128253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.129838 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:06.140036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.149796 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:06.254890 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.255838 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.258409 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.270936 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.271966 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:06.274822 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.275467 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:06.280747 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.281270 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.281483 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:06.284512 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.304638 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:06.453660 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:06.463647 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:06:06.498059 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:06.581611 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:06.631520 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.634284 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.634835 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.645118 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.649672 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:06.663333 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:06.667125 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.667521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.676434 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.678375 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:06.681723 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.690722 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:06.707091 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:06:06.729952 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:06.779533 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.780720 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.783335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:06.796923 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.804843 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:06.805559 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:06:06.815655 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:06.890555 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:06.929597 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:06.962009 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:06.983335 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:06.988822 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:06.994256 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:06.995955 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:07.007985 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:07.008615 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.015062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.016278 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:07.017755 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:07.043711 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:07.056165 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:07.107312 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 02:06:07.110594 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.111163 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.113869 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.115581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:07.116200 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.116317 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:07.216056 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.220444 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:07.221791 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.224885 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:07.225121 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:07.227944 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:07.232517 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:07.232647 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:07.234925 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:07.244256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.326261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.328943 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.329861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.340793 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.341835 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:07.346262 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:07.412175 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:07.510108 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:07.541699 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.541980 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.543504 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.550298 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:07.550815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:07.570086 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:07.587177 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:07.609191 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:07.661906 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.668422 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.670878 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.672425 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:07.683588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:07.683841 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.684006 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.687702 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:07.700270 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:07.723251 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:07.723539 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:07.726253 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:07.729604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:07.788761 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:06:07.893440 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 02:06:07.966576 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:08.043581 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.046545 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.047466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.049364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.050654 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.064092 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.075906 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.082125 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.086812 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:08.096192 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:08.101245 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.111136 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.116945 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:08.180867 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:08.203313 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.206573 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.207231 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.209771 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:08.218453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:08.223599 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:08.229347 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:08.320466 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:08.364178 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:08.393575 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.395251 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:08.395546 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:08.403074 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.403079 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.413122 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:08.420956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:08.466320 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:08.523954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.524967 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.530550 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.538944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.540329 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.542246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:08.577991 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:08.630605 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.640523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.644393 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:08.647926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.650113 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.668354 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.735028 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.736701 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.737777 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.742983 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.746307 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:08.750265 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:08.822849 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:08.880385 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:08.890619 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:08.961395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:08.963853 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.966457 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:08.969413 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:08.969601 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:08.975993 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:09.067442 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:09.095736 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.099890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.103865 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.115283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.116185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.117249 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:09.124596 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.128161 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:09.141302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.150391 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:09.151892 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.168224 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:09.235234 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:09.240362 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:09.306797 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:09.317181 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:09.332991 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:09.387722 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:09.414885 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:09.434303 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:09.457520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.458222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.464397 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.464793 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.474303 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:09.478266 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:09.492142 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:09.525065 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.526113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:09.534175 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:09.534893 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.538093 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.540033 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:09.545181 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:09.608399 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:06:09.628015 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:09.630839 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:09.633311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.635913 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.636860 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:09.639909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:09.655933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:09.675990 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:09.683709 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:06:09.705552 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:09.801713 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.804834 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.805222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.808003 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:09.810839 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:09.819250 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:09.848035 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:09.951992 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:09.952493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.954769 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:09.956214 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.958511 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:09.959688 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:09.966969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:10.026106 12841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:06:10.042301 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:10.057585 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.063948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:10.065511 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.072098 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:10.075856 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:10.077428 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.089974 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:10.170248 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.170785 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.173346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.176146 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:10.186267 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.189868 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:10.286391 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:10.380605 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.383927 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:10.384778 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.384788 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.408365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.409605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:10.418484 12843 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:06:10.505649 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:10.514909 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.515777 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.518959 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:10.525638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.530421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:10.542763 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:10.548993 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.555589 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.556002 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:10.566524 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:10.570456 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:10.576457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.576719 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:10.585947 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:10.783993 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:10.876891 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.877990 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.878037 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.878863 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.880888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:10.881529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:10.907035 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:10.948060 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:10.948683 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.949172 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.951630 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:10.952770 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:10.961176 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:10.976208 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.034324 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.036702 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.039939 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:11.043496 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:11.050184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.050918 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.065299 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:11.175201 13105 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:11.269610 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.269886 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.272774 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:11.273255 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:11.286840 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.287360 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.296247 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:11.352944 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:11.368345 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:11.373893 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.376271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.384284 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:11.385478 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.386204 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.386267 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:11.408143 12841 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:11.482877 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.484416 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.485415 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:11.497295 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.497712 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.505461 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.605988 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.606082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.606693 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.615937 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:11.618597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.625842 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:11.630095 12972 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:11.729454 13103 consensus_peers.cc:485] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:11.744664 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:11.746776 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:11.750649 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:06:11.760672 12974 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:06:11.768257 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:11.800117 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.802965 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.806594 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.817346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.817945 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.820276 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:11.843323 12974 consensus_peers.cc:462] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:11.879601 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:11.925153 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.925956 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:11.938769 12710 consensus_peers.cc:485] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:11.939190 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:11.940735 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:11.944561 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:11.952492 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.963433 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:11.967929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.972645 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:11.982025 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:11.983428 13105 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:12.001984 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.002046 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:12.054950 12712 consensus_peers.cc:485] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:06:12.113787 14738 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:12.175658 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:06:12.229646 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:12.229677 13105 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:06:12.257337 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:06:12.258095 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:06:12.317407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.318022 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.322378 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.327744 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.341221 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:12.341264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.341572 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.343329 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.350184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.357249 16318 raft_consensus.cc:998] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.305s)
I20240503 02:06:12.357668 16318 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:12.360766 12774 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:06:12.361719 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.362128 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:12.362372 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.363938 16318 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:12.378917 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.400681 16989 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:06:12.403170 12774 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:06:12.406216 12630 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
W20240503 02:06:12.417662 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:12.418534 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:12.475857 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.477365 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.479647 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.480785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.481266 12090 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:12.481364 12090 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:12.481732 12090 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:12.482497 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:06:12.484958 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:12.491868 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:12.491969 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:12.492103 12090 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:12.496196 12712 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:12.506536 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:12.600430 17040 raft_consensus.cc:998] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.097s)
I20240503 02:06:12.600883 17040 consensus_queue.cc:237] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:12.602358 14801 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:12.602389 13037 raft_consensus.cc:1261] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:12.602809 17040 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:12.602986 12090 consensus_queue.cc:1035] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:12.604959 17042 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.605579 14801 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.605703 13037 raft_consensus.cc:2932] T 6ad545d33bbc40458c75ed7c90d92ba7 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.608518 12629 catalog_manager.cc:5484] T 6ad545d33bbc40458c75ed7c90d92ba7 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
W20240503 02:06:12.641324 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:12.653767 16548 raft_consensus.cc:998] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.060s)
I20240503 02:06:12.654179 16548 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:12.655434 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:06:12.656237 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.656378 17063 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:12.658185 16548 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:06:12.658720 12907 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:06:12.660604 12627 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } }
I20240503 02:06:12.693626 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.695335 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.696259 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.698699 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.718283 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:12.719007 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.795203 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.796108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.802953 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.806713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.811913 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.837816 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:12.838266 14736 consensus_peers.cc:462] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:12.902279 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 02:06:12.914700 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.916114 16692 raft_consensus.cc:998] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.111s)
I20240503 02:06:12.916524 16692 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:12.916790 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.919169 14670 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:06:12.920126 17175 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:06:12.921514 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:12.921830 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:12.926070 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:12.926893 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:12.935328 17061 raft_consensus.cc:998] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.504s)
I20240503 02:06:12.935659 17061 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:12.937809 14801 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:06:12.938939 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:12.940757 16548 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:06:12.942586 17063 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.942958 14801 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.946285 12633 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:12.954277 17175 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:12.956983 14670 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:12.958959 12633 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 7 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:12.960361 13167 consensus_queue.cc:237] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:12.962172 14801 raft_consensus.cc:1261] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:06:12.962234 13104 consensus_peers.cc:485] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 141097686fe24c8f9956dcc1ad1d253c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.962798 17063 consensus_queue.cc:1035] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:12.966145 17063 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:12.966240 12907 consensus_queue.cc:237] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:12.967454 17061 raft_consensus.cc:998] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.093s)
I20240503 02:06:12.967710 17061 consensus_queue.cc:237] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:12.969017 13037 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:06:12.969950 16548 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:06:12.970995 14801 raft_consensus.cc:2932] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
W20240503 02:06:12.973683 12842 consensus_peers.cc:485] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.973623 12634 catalog_manager.cc:5484] T 141097686fe24c8f9956dcc1ad1d253c P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:12.975824 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 141097686fe24c8f9956dcc1ad1d253c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:12.976347 14670 raft_consensus.cc:1261] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:06:12.976538 17061 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.977061 17177 consensus_queue.cc:1035] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:12.978754 13037 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.979259 12633 catalog_manager.cc:5484] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 5 to 6, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.979702 12090 raft_consensus.cc:998] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.219s)
I20240503 02:06:12.980113 12090 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:06:12.981119 17177 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.981751 13037 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:06:12.982846 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 02:06:12.983019 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.983518 12625 catalog_manager.cc:5484] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.984838 13167 consensus_queue.cc:237] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:12.986310 14670 raft_consensus.cc:2932] T 0613ddd56ed346b3b6dfa9e7c3a4335e P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:12.987821 17040 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:12.988876 13037 raft_consensus.cc:1261] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:06:12.991652 13103 consensus_peers.cc:485] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 -> Peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): Couldn't send request to peer bbba999bf95446a68b6586608b33ee16. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:12.991909 17195 consensus_queue.cc:1035] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:12.993795 17040 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:12.993999 17195 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:12.995453 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0633596226954c439d2f671c25eebfe6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:12.996260 12634 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:12.996312 13038 raft_consensus.cc:2932] T 0633596226954c439d2f671c25eebfe6 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:12.997300 12625 catalog_manager.cc:5484] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:12.996315 13037 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:13.043469 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.043522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.048171 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.051894 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.054656 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.056728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:13.128604 12841 consensus_peers.cc:485] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:06:13.142917 8536 raft_consensus.cc:998] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.315s)
I20240503 02:06:13.143357 8536 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:13.144755 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:06:13.145169 8536 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:13.147112 17231 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.147591 13038 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.149820 12628 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 8 to 9, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.156080 14801 consensus_queue.cc:237] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:13.157557 13038 raft_consensus.cc:1261] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:06:13.157958 17231 consensus_queue.cc:1035] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:13.159605 8536 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:13.160002 13038 raft_consensus.cc:2932] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:06:13.161358 14737 consensus_peers.cc:485] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.162550 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 063a7f2ed28a455697b000f52a4343a3 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:13.162776 12628 catalog_manager.cc:5484] T 063a7f2ed28a455697b000f52a4343a3 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 9 to 10, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:13.237877 17235 raft_consensus.cc:998] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.005s)
I20240503 02:06:13.238247 17235 consensus_queue.cc:237] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.241235 12907 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:06:13.241423 14665 raft_consensus.cc:1261] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:06:13.241780 8536 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:13.242374 8536 consensus_queue.cc:1035] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:13.246201 8536 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.246534 14665 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.249130 12630 catalog_manager.cc:5484] T 007feed148fb44e9a6fe9d449b6d8f9d P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.256017 12907 raft_consensus.cc:2932] T 007feed148fb44e9a6fe9d449b6d8f9d P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.265717 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.268334 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.269731 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.279892 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.281767 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.282392 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:13.392966 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.400446 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:13.408771 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:13.414320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.415906 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.429351 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.430208 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.430792 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.431478 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.452298 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.452653 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.460615 17372 ts_tablet_manager.cc:902] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:06:13.461117 17372 tablet_copy_client.cc:308] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
W20240503 02:06:13.468417 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 007feed148fb44e9a6fe9d449b6d8f9d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:13.476572 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.484377 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 141097686fe24c8f9956dcc1ad1d253c from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:38387)
I20240503 02:06:13.484516 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 141097686fe24c8f9956dcc1ad1d253c from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:38387: session id = d23425fa2b5848c8b1a2b737d75e69eb-141097686fe24c8f9956dcc1ad1d253c
I20240503 02:06:13.486004 13189 tablet_copy_source_session.cc:205] T 141097686fe24c8f9956dcc1ad1d253c P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:13.487955 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141097686fe24c8f9956dcc1ad1d253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:13.490147 17372 tablet_copy_client.cc:783] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:13.495095 17372 tablet_copy_client.cc:647] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:13.496269 17372 tablet_copy_client.cc:523] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:13.497678 17372 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:06:13.507198 17372 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:13.507539 17372 tablet_bootstrap.cc:492] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:06:13.507769 17372 ts_tablet_manager.cc:1367] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:06:13.508152 17372 raft_consensus.cc:348] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.508252 17372 raft_consensus.cc:729] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:06:13.508365 17372 consensus_queue.cc:260] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.508635 17372 ts_tablet_manager.cc:1398] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:06:13.509214 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-141097686fe24c8f9956dcc1ad1d253c received from {username='slave'} at 127.0.99.134:38387
I20240503 02:06:13.509299 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-141097686fe24c8f9956dcc1ad1d253c on tablet 141097686fe24c8f9956dcc1ad1d253c with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:13.536640 17379 ts_tablet_manager.cc:902] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:06:13.537200 17379 tablet_copy_client.cc:308] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:06:13.537706 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:48855)
I20240503 02:06:13.537808 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:48855: session id = 03ced8676740485e86ebe292f8ee4175-0613ddd56ed346b3b6dfa9e7c3a4335e
I20240503 02:06:13.538959 12927 tablet_copy_source_session.cc:205] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:13.539827 17379 ts_tablet_manager.cc:1875] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:13.541661 17379 ts_tablet_manager.cc:1888] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:06:13.541826 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0613ddd56ed346b3b6dfa9e7c3a4335e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:13.542934 17379 tablet_copy_client.cc:783] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:13.547879 17379 tablet_copy_client.cc:647] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:13.549124 17379 tablet_copy_client.cc:523] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:13.550369 17379 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:06:13.557600 17385 ts_tablet_manager.cc:902] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:06:13.557968 17379 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:13.558219 17385 tablet_copy_client.cc:308] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:06:13.558259 17379 tablet_bootstrap.cc:492] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:06:13.558437 17379 ts_tablet_manager.cc:1367] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:06:13.558851 17379 raft_consensus.cc:348] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:13.558954 17379 raft_consensus.cc:729] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:06:13.559096 17379 consensus_queue.cc:260] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:13.559358 17379 ts_tablet_manager.cc:1398] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:06:13.559906 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-0613ddd56ed346b3b6dfa9e7c3a4335e received from {username='slave'} at 127.0.99.132:48855
I20240503 02:06:13.559995 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-0613ddd56ed346b3b6dfa9e7c3a4335e on tablet 0613ddd56ed346b3b6dfa9e7c3a4335e with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:06:13.562427 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 0633596226954c439d2f671c25eebfe6 from peer bbba999bf95446a68b6586608b33ee16 ({username='slave'} at 127.0.99.129:43941)
I20240503 02:06:13.562516 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 0633596226954c439d2f671c25eebfe6 from peer bbba999bf95446a68b6586608b33ee16 at {username='slave'} at 127.0.99.129:43941: session id = bbba999bf95446a68b6586608b33ee16-0633596226954c439d2f671c25eebfe6
I20240503 02:06:13.563298 13189 tablet_copy_source_session.cc:205] T 0633596226954c439d2f671c25eebfe6 P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:13.564113 17385 ts_tablet_manager.cc:1875] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:13.565882 17385 ts_tablet_manager.cc:1888] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:06:13.566031 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0633596226954c439d2f671c25eebfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:13.566927 17385 tablet_copy_client.cc:783] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:13.571439 17385 tablet_copy_client.cc:647] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:13.572656 17385 tablet_copy_client.cc:523] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:13.573827 17385 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:06:13.580933 17385 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:13.581175 17385 tablet_bootstrap.cc:492] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Bootstrap complete.
I20240503 02:06:13.581326 17385 ts_tablet_manager.cc:1367] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:06:13.581691 17385 raft_consensus.cc:348] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:13.581795 17385 raft_consensus.cc:729] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:06:13.581918 17385 consensus_queue.cc:260] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:13.582185 17385 ts_tablet_manager.cc:1398] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:06:13.582734 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session bbba999bf95446a68b6586608b33ee16-0633596226954c439d2f671c25eebfe6 received from {username='slave'} at 127.0.99.129:43941
I20240503 02:06:13.582810 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session bbba999bf95446a68b6586608b33ee16-0633596226954c439d2f671c25eebfe6 on tablet 0633596226954c439d2f671c25eebfe6 with peer bbba999bf95446a68b6586608b33ee16
W20240503 02:06:13.600498 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:06:13.646569 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e): 4838 bytes on disk
I20240503 02:06:13.647014 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(0613ddd56ed346b3b6dfa9e7c3a4335e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:06:13.659538 17379 ts_tablet_manager.cc:902] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:06:13.660068 17379 tablet_copy_client.cc:308] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37171
I20240503 02:06:13.660531 14822 tablet_copy_service.cc:140] P 754fc04ea9914073a56c0cf164f63240: Received BeginTabletCopySession request for tablet 063a7f2ed28a455697b000f52a4343a3 from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:35815)
I20240503 02:06:13.660634 14822 tablet_copy_service.cc:161] P 754fc04ea9914073a56c0cf164f63240: Beginning new tablet copy session on tablet 063a7f2ed28a455697b000f52a4343a3 from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:35815: session id = 03ced8676740485e86ebe292f8ee4175-063a7f2ed28a455697b000f52a4343a3
I20240503 02:06:13.661859 14822 tablet_copy_source_session.cc:205] T 063a7f2ed28a455697b000f52a4343a3 P 754fc04ea9914073a56c0cf164f63240: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:13.662644 17379 ts_tablet_manager.cc:1875] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:13.664451 17379 ts_tablet_manager.cc:1888] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240503 02:06:13.664623 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 063a7f2ed28a455697b000f52a4343a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:13.665453 17379 tablet_copy_client.cc:783] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:13.670102 17379 tablet_copy_client.cc:647] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:13.671341 17379 tablet_copy_client.cc:523] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:13.672559 17379 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:06:13.673020 17063 raft_consensus.cc:998] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.061s)
I20240503 02:06:13.673382 17063 consensus_queue.cc:237] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:13.674580 14801 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:13.674650 12907 raft_consensus.cc:1261] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:13.674919 17063 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:13.675057 17061 consensus_queue.cc:1035] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:13.677428 17061 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.677767 12907 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.679395 14801 raft_consensus.cc:2932] T 097947dce87b409ca4a57cb752fdd5e2 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.681084 12632 catalog_manager.cc:5484] T 097947dce87b409ca4a57cb752fdd5e2 P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.681747 17379 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:13.682067 17379 tablet_bootstrap.cc:492] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:06:13.682233 17379 ts_tablet_manager.cc:1367] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 02:06:13.683305 17379 raft_consensus.cc:348] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:13.683403 17379 raft_consensus.cc:729] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:06:13.683560 17379 consensus_queue.cc:260] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:13.683866 17379 ts_tablet_manager.cc:1398] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:06:13.684373 14822 tablet_copy_service.cc:342] P 754fc04ea9914073a56c0cf164f63240: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-063a7f2ed28a455697b000f52a4343a3 received from {username='slave'} at 127.0.99.132:35815
I20240503 02:06:13.684477 14822 tablet_copy_service.cc:433] P 754fc04ea9914073a56c0cf164f63240: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-063a7f2ed28a455697b000f52a4343a3 on tablet 063a7f2ed28a455697b000f52a4343a3 with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:06:13.696981 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6): 4838 bytes on disk
I20240503 02:06:13.697485 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(0633596226954c439d2f671c25eebfe6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240503 02:06:13.738669 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c): 4922 bytes on disk
I20240503 02:06:13.739243 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(141097686fe24c8f9956dcc1ad1d253c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 02:06:13.759125 12090 raft_consensus.cc:998] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.294s)
I20240503 02:06:13.759547 12090 consensus_queue.cc:237] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:13.761655 14801 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:13.762032 12090 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:13.762504 13038 raft_consensus.cc:1261] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:13.763886 12090 consensus_queue.cc:1035] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:13.764413 12090 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.766968 14801 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 754fc04ea9914073a56c0cf164f63240 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.767719 12629 catalog_manager.cc:5484] T 3b829f779c8a459597c47dc3e42a4c46 P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.771929 13038 raft_consensus.cc:2932] T 3b829f779c8a459597c47dc3e42a4c46 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:13.786438 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.798561 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.799644 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.804380 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.809784 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.818291 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.822768 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.823379 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:13.848654 17371 raft_consensus.cc:998] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.029s)
I20240503 02:06:13.849004 17371 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.851974 14665 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:06:13.854158 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.854797 16989 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:06:13.857507 16989 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.860219 12627 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.862082 14665 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.866914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:13.868172 13038 consensus_queue.cc:237] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:13.872502 14665 raft_consensus.cc:1203] T 141097686fe24c8f9956dcc1ad1d253c P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 02:06:13.876715 12972 consensus_peers.cc:485] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 -> Peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): Couldn't send request to peer bbba999bf95446a68b6586608b33ee16. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 613074d3c4514bd5b64a06c59af09a56. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.878031 14665 raft_consensus.cc:1261] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:06:13.878394 16989 consensus_queue.cc:1035] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:13.880944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.882000 16989 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:13.882409 14665 raft_consensus.cc:2932] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:13.883661 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 613074d3c4514bd5b64a06c59af09a56 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:13.885010 12627 catalog_manager.cc:5484] T 613074d3c4514bd5b64a06c59af09a56 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 8 to 9, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:13.894585 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:13.898394 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3): 4923 bytes on disk
I20240503 02:06:13.898862 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(063a7f2ed28a455697b000f52a4343a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 02:06:13.933384 17499 raft_consensus.cc:998] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.324s)
I20240503 02:06:13.934005 17499 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.935370 14665 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:06:13.935810 17175 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:06:13.937604 17499 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.938163 14665 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.940390 12625 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:13.944504 12907 consensus_queue.cc:237] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
W20240503 02:06:13.947674 12842 consensus_peers.cc:485] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:13.947726 14665 raft_consensus.cc:1261] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:06:13.948110 17175 consensus_queue.cc:1035] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:13.949491 17499 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:13.949880 14665 raft_consensus.cc:2932] T 490776bd408a48a9bcbbc7a501baa8ce P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:13.951112 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 490776bd408a48a9bcbbc7a501baa8ce with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:13.952209 12625 catalog_manager.cc:5484] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:13.987289 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.988283 17492 raft_consensus.cc:998] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.183s)
I20240503 02:06:13.988623 17492 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:13.989979 14665 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:06:13.990842 17371 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:13.991345 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:13.992738 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:13.992974 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.002902 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.003454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.012662 17371 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.015106 14665 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.015540 12628 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 9 to 10, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.019649 13038 consensus_queue.cc:237] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:06:14.020969 14665 raft_consensus.cc:1261] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:06:14.021160 12973 consensus_peers.cc:485] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 -> Peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951): Couldn't send request to peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:14.021354 17492 consensus_queue.cc:1035] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:06:14.022893 16989 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:06:14.023293 14665 raft_consensus.cc:2932] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:06:14.024883 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a105660c12094c71b3a4b80e661d9308 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:14.025645 12626 catalog_manager.cc:5484] T a105660c12094c71b3a4b80e661d9308 P d23425fa2b5848c8b1a2b737d75e69eb reported cstate change: config changed from index 10 to 11, VOTER 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
W20240503 02:06:14.039031 14738 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:06:14.080015 12774 raft_consensus.cc:1203] T 0633596226954c439d2f671c25eebfe6 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:06:14.098171 13167 raft_consensus.cc:1203] T 0613ddd56ed346b3b6dfa9e7c3a4335e P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:06:14.144342 17175 raft_consensus.cc:998] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.236s)
I20240503 02:06:14.144757 17175 consensus_queue.cc:237] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:14.146204 13038 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:14.146204 14665 raft_consensus.cc:1261] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Refusing update from remote peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:06:14.146600 17507 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:14.146749 17175 consensus_queue.cc:1035] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:06:14.148552 17507 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.148914 13038 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P 22b582a5f4774861a5619c51198d39c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.150810 14665 raft_consensus.cc:2932] T da7d3c59eeb949ec84db013f24e656b0 P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.153030 12628 catalog_manager.cc:5484] T da7d3c59eeb949ec84db013f24e656b0 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 6 to 7, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.176342 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.176939 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.178222 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.182490 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.185109 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.194404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.258147 17594 ts_tablet_manager.cc:902] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:06:14.258759 17594 tablet_copy_client.cc:308] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:06:14.259279 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 613074d3c4514bd5b64a06c59af09a56 from peer bbba999bf95446a68b6586608b33ee16 ({username='slave'} at 127.0.99.129:51779)
I20240503 02:06:14.259402 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 613074d3c4514bd5b64a06c59af09a56 from peer bbba999bf95446a68b6586608b33ee16 at {username='slave'} at 127.0.99.129:51779: session id = bbba999bf95446a68b6586608b33ee16-613074d3c4514bd5b64a06c59af09a56
I20240503 02:06:14.260792 13055 tablet_copy_source_session.cc:205] T 613074d3c4514bd5b64a06c59af09a56 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.261725 17594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613074d3c4514bd5b64a06c59af09a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.263604 17594 tablet_copy_client.cc:783] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.268656 17594 tablet_copy_client.cc:647] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.269970 17594 tablet_copy_client.cc:523] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.271544 17594 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:06:14.280457 17594 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.280831 17594 tablet_bootstrap.cc:492] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Bootstrap complete.
I20240503 02:06:14.281054 17594 ts_tablet_manager.cc:1367] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 02:06:14.281504 17594 raft_consensus.cc:348] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.281628 17594 raft_consensus.cc:729] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.281780 17594 consensus_queue.cc:260] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.282307 13167 raft_consensus.cc:1203] T 063a7f2ed28a455697b000f52a4343a3 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 02:06:14.282086 17594 ts_tablet_manager.cc:1398] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:06:14.283669 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session bbba999bf95446a68b6586608b33ee16-613074d3c4514bd5b64a06c59af09a56 received from {username='slave'} at 127.0.99.129:51779
I20240503 02:06:14.283784 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session bbba999bf95446a68b6586608b33ee16-613074d3c4514bd5b64a06c59af09a56 on tablet 613074d3c4514bd5b64a06c59af09a56 with peer bbba999bf95446a68b6586608b33ee16
I20240503 02:06:14.321338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.321609 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.322867 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.324275 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.325454 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.326778 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:14.330543 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:14.354573 17371 raft_consensus.cc:998] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.136s)
I20240503 02:06:14.354910 17371 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:06:14.355998 12907 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 02:06:14.356355 17492 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:06:14.358377 17492 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 5 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:14.360149 12907 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 5 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:14.362105 12628 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 5 to 8, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:06:14.365399 13038 consensus_queue.cc:237] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:14.367019 12907 raft_consensus.cc:1261] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:06:14.367453 16989 consensus_queue.cc:1035] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:14.370265 16989 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:14.371825 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 34e88e69beb64036bb676b771dabd2a4 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:14.371973 12907 raft_consensus.cc:2932] T 34e88e69beb64036bb676b771dabd2a4 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:06:14.372184 12973 consensus_peers.cc:485] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34e88e69beb64036bb676b771dabd2a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:14.372766 12631 catalog_manager.cc:5484] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:06:14.399019 17395 raft_consensus.cc:998] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.499s)
I20240503 02:06:14.399392 17395 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:14.400626 14665 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:06:14.401060 17395 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:06:14.402837 17631 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.404377 14665 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.405280 12628 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 8 to 9, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:06:14.408447 13167 consensus_queue.cc:237] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
W20240503 02:06:14.409883 13103 consensus_peers.cc:485] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 -> Peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): Couldn't send request to peer bbba999bf95446a68b6586608b33ee16. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:14.410363 14665 raft_consensus.cc:1261] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:06:14.410760 17631 consensus_queue.cc:1035] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:14.412693 17395 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:14.413066 14665 raft_consensus.cc:2932] T 8292c06377f445849acdc22d08bd612d P d23425fa2b5848c8b1a2b737d75e69eb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:14.415467 12628 catalog_manager.cc:5484] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 9 to 10, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:14.415990 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8292c06377f445849acdc22d08bd612d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:14.443930 17668 ts_tablet_manager.cc:902] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a (127.0.99.130:43951)
I20240503 02:06:14.444753 17668 tablet_copy_client.cc:308] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:43951
I20240503 02:06:14.445204 12927 tablet_copy_service.cc:140] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Received BeginTabletCopySession request for tablet 490776bd408a48a9bcbbc7a501baa8ce from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:48855)
I20240503 02:06:14.445307 12927 tablet_copy_service.cc:161] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Beginning new tablet copy session on tablet 490776bd408a48a9bcbbc7a501baa8ce from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:48855: session id = 03ced8676740485e86ebe292f8ee4175-490776bd408a48a9bcbbc7a501baa8ce
I20240503 02:06:14.446333 12927 tablet_copy_source_session.cc:205] T 490776bd408a48a9bcbbc7a501baa8ce P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.448746 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.450762 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.450891 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56): 4584 bytes on disk
I20240503 02:06:14.451341 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(613074d3c4514bd5b64a06c59af09a56) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.451462 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.452594 17668 ts_tablet_manager.cc:1875] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:14.457159 17668 ts_tablet_manager.cc:1888] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:06:14.457365 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490776bd408a48a9bcbbc7a501baa8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.458617 17668 tablet_copy_client.cc:783] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.463701 17668 tablet_copy_client.cc:647] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.464946 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.465855 17668 tablet_copy_client.cc:523] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.467191 17668 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:06:14.467546 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7309 bytes on disk
I20240503 02:06:14.470211 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"thread_start_us":80,"threads_started":1}
W20240503 02:06:14.471966 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 34e88e69beb64036bb676b771dabd2a4: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:14.475833 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.485915 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.487154 17668 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.487496 17668 tablet_bootstrap.cc:492] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:06:14.487722 17668 ts_tablet_manager.cc:1367] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.006s
I20240503 02:06:14.488229 17668 raft_consensus.cc:348] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:14.488377 17668 raft_consensus.cc:729] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.488545 17668 consensus_queue.cc:260] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:14.489328 17668 ts_tablet_manager.cc:1398] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:06:14.489933 12927 tablet_copy_service.cc:342] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-490776bd408a48a9bcbbc7a501baa8ce received from {username='slave'} at 127.0.99.132:48855
I20240503 02:06:14.490016 12927 tablet_copy_service.cc:433] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-490776bd408a48a9bcbbc7a501baa8ce on tablet 490776bd408a48a9bcbbc7a501baa8ce with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:06:14.514916 17234 raft_consensus.cc:998] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Attempting to remove follower 9e39ab9b64544988b968015fe9fa06e6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9e39ab9b64544988b968015fe9fa06e6 for more than 300 seconds (300.468s)
I20240503 02:06:14.515311 17234 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } }
I20240503 02:06:14.516631 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:06:14.517083 17693 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:06:14.518961 17234 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:14.519465 13038 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:14.522259 12628 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 8 to 10, VOTER 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133) evicted. New cstate: current_term: 3 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } }
I20240503 02:06:14.526223 14801 consensus_queue.cc:237] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.527451 13038 raft_consensus.cc:1261] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Refusing update from remote peer 754fc04ea9914073a56c0cf164f63240: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:06:14.527855 17693 consensus_queue.cc:1035] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:06:14.530225 17693 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
W20240503 02:06:14.531234 14736 consensus_peers.cc:485] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 -> Peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): Couldn't send request to peer bbba999bf95446a68b6586608b33ee16. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:06:14.531890 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f5767bdfcc6249578f7c3ada3c08da6e with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:06:14.532184 13038 raft_consensus.cc:2932] T f5767bdfcc6249578f7c3ada3c08da6e P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:14.532933 12628 catalog_manager.cc:5484] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240 reported cstate change: config changed from index 10 to 11, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) added. New cstate: current_term: 3 leader_uuid: "754fc04ea9914073a56c0cf164f63240" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } }
I20240503 02:06:14.566183 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.567931 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.568499 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.569342 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.580049 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.594774 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.641597 17728 ts_tablet_manager.cc:902] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:06:14.642148 17728 tablet_copy_client.cc:308] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:06:14.642704 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet a105660c12094c71b3a4b80e661d9308 from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a ({username='slave'} at 127.0.99.130:40943)
I20240503 02:06:14.642814 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet a105660c12094c71b3a4b80e661d9308 from peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a at {username='slave'} at 127.0.99.130:40943: session id = 96b8f7d7c1ce4e2cad5af6caabfd8c4a-a105660c12094c71b3a4b80e661d9308
I20240503 02:06:14.643685 13055 tablet_copy_source_session.cc:205] T a105660c12094c71b3a4b80e661d9308 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.644819 17728 ts_tablet_manager.cc:1875] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:14.646723 17728 ts_tablet_manager.cc:1888] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:06:14.646891 17728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a105660c12094c71b3a4b80e661d9308. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.647879 17728 tablet_copy_client.cc:783] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.652293 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce): 4581 bytes on disk
I20240503 02:06:14.652403 17728 tablet_copy_client.cc:647] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.652751 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(490776bd408a48a9bcbbc7a501baa8ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.653720 17728 tablet_copy_client.cc:523] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.655119 17728 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap starting.
I20240503 02:06:14.664927 17728 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.665196 17728 tablet_bootstrap.cc:492] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Bootstrap complete.
I20240503 02:06:14.665364 17728 ts_tablet_manager.cc:1367] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.006s
I20240503 02:06:14.665729 17728 raft_consensus.cc:348] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:06:14.665834 17728 raft_consensus.cc:729] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b8f7d7c1ce4e2cad5af6caabfd8c4a, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.665972 17728 consensus_queue.cc:260] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:06:14.666208 17728 ts_tablet_manager.cc:1398] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:06:14.666738 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 96b8f7d7c1ce4e2cad5af6caabfd8c4a-a105660c12094c71b3a4b80e661d9308 received from {username='slave'} at 127.0.99.130:40943
I20240503 02:06:14.666812 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 96b8f7d7c1ce4e2cad5af6caabfd8c4a-a105660c12094c71b3a4b80e661d9308 on tablet a105660c12094c71b3a4b80e661d9308 with peer 96b8f7d7c1ce4e2cad5af6caabfd8c4a
W20240503 02:06:14.668774 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:06:14.761477 12774 raft_consensus.cc:1203] T 613074d3c4514bd5b64a06c59af09a56 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:06:14.785418 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.786157 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.786399 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.787140 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.792846 17668 ts_tablet_manager.cc:902] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:06:14.793289 17668 tablet_copy_client.cc:308] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:06:14.794175 12954 maintenance_manager.cc:419] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: Scheduling UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308): 4923 bytes on disk
I20240503 02:06:14.794929 12844 maintenance_manager.cc:643] P 96b8f7d7c1ce4e2cad5af6caabfd8c4a: UndoDeltaBlockGCOp(a105660c12094c71b3a4b80e661d9308) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.795284 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet 34e88e69beb64036bb676b771dabd2a4 from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:57713)
I20240503 02:06:14.795382 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet 34e88e69beb64036bb676b771dabd2a4 from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:57713: session id = 03ced8676740485e86ebe292f8ee4175-34e88e69beb64036bb676b771dabd2a4
I20240503 02:06:14.796416 13055 tablet_copy_source_session.cc:205] T 34e88e69beb64036bb676b771dabd2a4 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.797438 17668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e88e69beb64036bb676b771dabd2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.800271 17668 tablet_copy_client.cc:783] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.800312 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.804649 17780 ts_tablet_manager.cc:902] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:06:14.805182 17780 tablet_copy_client.cc:308] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:06:14.808245 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 8292c06377f445849acdc22d08bd612d from peer bbba999bf95446a68b6586608b33ee16 ({username='slave'} at 127.0.99.129:43941)
I20240503 02:06:14.808339 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 8292c06377f445849acdc22d08bd612d from peer bbba999bf95446a68b6586608b33ee16 at {username='slave'} at 127.0.99.129:43941: session id = bbba999bf95446a68b6586608b33ee16-8292c06377f445849acdc22d08bd612d
I20240503 02:06:14.809319 13189 tablet_copy_source_session.cc:205] T 8292c06377f445849acdc22d08bd612d P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.810055 17780 ts_tablet_manager.cc:1875] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:14.811944 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.822582 17780 ts_tablet_manager.cc:1888] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240503 02:06:14.823186 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8292c06377f445849acdc22d08bd612d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.825369 17780 tablet_copy_client.cc:783] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.839687 17780 tablet_copy_client.cc:647] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.839979 17668 tablet_copy_client.cc:647] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.841274 17668 tablet_copy_client.cc:523] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.842736 17668 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:06:14.844640 17780 tablet_copy_client.cc:523] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.845902 17780 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:06:14.852857 17668 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.853765 17780 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.854064 17780 tablet_bootstrap.cc:492] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Bootstrap complete.
I20240503 02:06:14.854249 17780 ts_tablet_manager.cc:1367] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.006s
I20240503 02:06:14.854701 17780 raft_consensus.cc:348] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.854871 17780 raft_consensus.cc:729] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.854842 17668 tablet_bootstrap.cc:492] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:06:14.855082 17668 ts_tablet_manager.cc:1367] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:06:14.855584 17668 raft_consensus.cc:348] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:14.855790 17668 raft_consensus.cc:729] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.855077 17780 consensus_queue.cc:260] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.857100 17780 ts_tablet_manager.cc:1398] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:06:14.857828 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session bbba999bf95446a68b6586608b33ee16-8292c06377f445849acdc22d08bd612d received from {username='slave'} at 127.0.99.129:43941
I20240503 02:06:14.857899 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session bbba999bf95446a68b6586608b33ee16-8292c06377f445849acdc22d08bd612d on tablet 8292c06377f445849acdc22d08bd612d with peer bbba999bf95446a68b6586608b33ee16
I20240503 02:06:14.860400 17668 consensus_queue.cc:260] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:06:14.860731 17668 ts_tablet_manager.cc:1398] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:06:14.861187 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-34e88e69beb64036bb676b771dabd2a4 received from {username='slave'} at 127.0.99.132:57713
I20240503 02:06:14.861274 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-34e88e69beb64036bb676b771dabd2a4 on tablet 34e88e69beb64036bb676b771dabd2a4 with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:06:14.904458 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4): 4923 bytes on disk
I20240503 02:06:14.905108 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(34e88e69beb64036bb676b771dabd2a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.905380 17780 ts_tablet_manager.cc:902] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Initiating tablet copy from peer 754fc04ea9914073a56c0cf164f63240 (127.0.99.135:37171)
I20240503 02:06:14.905805 17780 tablet_copy_client.cc:308] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:37171
I20240503 02:06:14.906826 14822 tablet_copy_service.cc:140] P 754fc04ea9914073a56c0cf164f63240: Received BeginTabletCopySession request for tablet f5767bdfcc6249578f7c3ada3c08da6e from peer bbba999bf95446a68b6586608b33ee16 ({username='slave'} at 127.0.99.129:36657)
I20240503 02:06:14.906941 14822 tablet_copy_service.cc:161] P 754fc04ea9914073a56c0cf164f63240: Beginning new tablet copy session on tablet f5767bdfcc6249578f7c3ada3c08da6e from peer bbba999bf95446a68b6586608b33ee16 at {username='slave'} at 127.0.99.129:36657: session id = bbba999bf95446a68b6586608b33ee16-f5767bdfcc6249578f7c3ada3c08da6e
I20240503 02:06:14.908097 14822 tablet_copy_source_session.cc:205] T f5767bdfcc6249578f7c3ada3c08da6e P 754fc04ea9914073a56c0cf164f63240: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:06:14.909158 17780 ts_tablet_manager.cc:1875] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:06:14.911736 17780 ts_tablet_manager.cc:1888] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240503 02:06:14.911947 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5767bdfcc6249578f7c3ada3c08da6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:06:14.913064 17780 tablet_copy_client.cc:783] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 5 data blocks...
I20240503 02:06:14.921722 17780 tablet_copy_client.cc:647] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet copy: Starting download of 1 WAL segments...
I20240503 02:06:14.923619 17780 tablet_copy_client.cc:523] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:06:14.925099 17780 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Bootstrap starting.
I20240503 02:06:14.935654 17780 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:06:14.935999 17780 tablet_bootstrap.cc:492] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Bootstrap complete.
I20240503 02:06:14.936712 17780 ts_tablet_manager.cc:1367] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.002s
I20240503 02:06:14.937280 17780 raft_consensus.cc:348] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.937407 17780 raft_consensus.cc:729] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Initialized, Role: FOLLOWER
I20240503 02:06:14.937570 17780 consensus_queue.cc:260] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } }
I20240503 02:06:14.937852 17780 ts_tablet_manager.cc:1398] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:06:14.938374 14822 tablet_copy_service.cc:342] P 754fc04ea9914073a56c0cf164f63240: Request end of tablet copy session bbba999bf95446a68b6586608b33ee16-f5767bdfcc6249578f7c3ada3c08da6e received from {username='slave'} at 127.0.99.129:36657
I20240503 02:06:14.938458 14822 tablet_copy_service.cc:433] P 754fc04ea9914073a56c0cf164f63240: ending tablet copy session bbba999bf95446a68b6586608b33ee16-f5767bdfcc6249578f7c3ada3c08da6e on tablet f5767bdfcc6249578f7c3ada3c08da6e with peer bbba999bf95446a68b6586608b33ee16
I20240503 02:06:14.943789 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.950377 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.953991 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d): 4583 bytes on disk
I20240503 02:06:14.954449 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.954413 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(8292c06377f445849acdc22d08bd612d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.955389 12823 maintenance_manager.cc:419] P bbba999bf95446a68b6586608b33ee16: Scheduling UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e): 4583 bytes on disk
I20240503 02:06:14.955860 12713 maintenance_manager.cc:643] P bbba999bf95446a68b6586608b33ee16: UndoDeltaBlockGCOp(f5767bdfcc6249578f7c3ada3c08da6e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:06:14.964267 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.965775 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.966329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:14.968430 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:14.973908 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.979296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:14.980376 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:14.993508 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:14.999112 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:06:15.001987 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:15.009166 13167 raft_consensus.cc:1203] T 490776bd408a48a9bcbbc7a501baa8ce P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:06:15.025544 12905 raft_consensus.cc:1203] T a105660c12094c71b3a4b80e661d9308 P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 02:06:15.072003 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:06:15.191929 13167 raft_consensus.cc:1203] T 34e88e69beb64036bb676b771dabd2a4 P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240503 02:06:15.216959 12774 raft_consensus.cc:1203] T 8292c06377f445849acdc22d08bd612d P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 02:06:15.310787 12774 raft_consensus.cc:1203] T f5767bdfcc6249578f7c3ada3c08da6e P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240503 02:06:15.324576 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.326988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.327734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.336874 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:15.340646 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.342114 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.362371 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.367376 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.371910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.382889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.383821 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.384135 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:15.400691 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:06:15.507076 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:06:15.507369 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.508564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.509163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.511687 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.513720 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.525611 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:15.566701 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 007feed148fb44e9a6fe9d449b6d8f9d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:15.713917 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.714789 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.716094 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.719671 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.733132 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:15.733480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.819694 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.823231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.834353 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.835222 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.846935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.847117 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:15.933614 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.933885 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:15.934254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:15.937359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:15.951715 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:15.953224 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.044283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.048254 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.053335 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.055912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.056317 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.061812 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:16.189019 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:06:16.259161 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.259443 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.262135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.262778 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.270210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.288483 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.376490 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.383170 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.383229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.394603 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.398150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.406883 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.408246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.429030 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.438613 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.443485 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.443861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.444041 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:16.596671 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8292c06377f445849acdc22d08bd612d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:16.737586 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.738768 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.739847 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.748557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:16.749284 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:06:16.752990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:16.761910 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 02:06:16.770162 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.813243 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.813334 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:16.815588 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.825089 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.825803 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.826629 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.920456 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:16.922051 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.922750 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.927774 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:16.945458 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:16.948724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:17.069094 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.071565 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.077868 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.078537 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.081673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:17.096405 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:17.205250 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:06:17.242373 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.245011 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.245894 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:17.252153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.253906 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.255535 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:17.272562 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:06:17.281817 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 36 similar messages]
I20240503 02:06:17.356498 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.358348 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.359485 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:17.364027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.369002 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.371112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:17.391786 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:06:17.448258 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.451076 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.452844 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.457405 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:17.458294 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 150 similar messages]
I20240503 02:06:17.462628 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.466928 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:17.663551 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 007feed148fb44e9a6fe9d449b6d8f9d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:17.674453 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.675057 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.677400 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:17.678246 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.678901 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.696471 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:17.794775 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.797560 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.802277 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.808138 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.811524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:17.819087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.829825 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:17.830192 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:17.836043 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 02:06:17.839011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:17.844272 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:17.857707 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:17.874464 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:17.933058 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:06:17.990027 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240503 02:06:18.152706 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.156180 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.165633 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.167445 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.172267 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.185403 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.228042 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.231387 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.232427 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.232447 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.236732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.249624 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.256263 18580 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:18.256377 18580 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:18.256762 18580 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:06:18.258212 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:18.258293 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:18.258373 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:18.258499 18580 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:18.337934 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.341722 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.343914 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.344051 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.346061 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.358546 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.489562 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.492555 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.495996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.497866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.499929 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.531246 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:18.640197 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 02:06:18.653437 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.654932 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.655498 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.656493 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.664150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:18.678798 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8292c06377f445849acdc22d08bd612d: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:18.680382 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.768780 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.770748 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.771315 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:18.773545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.787541 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.796262 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.862490 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:18.863219 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.867367 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.873037 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:18.874051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:18.878163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.104694 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.107221 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.108407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.114326 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.115063 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:19.126451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.224720 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.234871 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.236136 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.243137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.249819 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.253295 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.259004 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.261343 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.265154 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:19.266935 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.272003 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.296350 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:19.340997 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:06:19.617064 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.618465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.620283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.620657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.631728 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.632962 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.641009 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.647086 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.651904 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:19.662999 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.667529 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:19.680703 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:06:19.682893 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.766763 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.769186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.769508 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.771415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.772948 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:19.785825 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:19.955822 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:06:19.983255 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:19.986872 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:19.987953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:19.988794 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:19.992848 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:06:19.999541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.010514 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.098976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.103875 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.105209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.105238 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:20.112962 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.128821 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.210242 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.212312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.214955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:20.215924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.225924 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.234992 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:20.294354 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:06:20.317032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.317638 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.318220 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.325273 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:20.331730 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.336309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:20.381186 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 02:06:20.532933 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.533752 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.534304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.536502 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:20.538551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.546878 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.655912 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.664427 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.664445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.665870 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.683806 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.683836 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:20.689478 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.694613 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.707809 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:20.712164 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:20.712595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:20.728365 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:20.847743 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:21.027810 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.030305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.036579 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.048527 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.048853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.064649 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.075145 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.080294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.081460 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.088843 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.098776 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.108695 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:21.126051 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 02:06:21.197178 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.198936 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.199379 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.200117 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.210583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.220937 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.396293 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.398474 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.402277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.404630 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.412863 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.430965 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.514561 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.517113 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.519512 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.520403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.521759 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.521816 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.619290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.620196 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.625615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.628008 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.628687 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.629359 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:21.657717 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 02:06:21.724983 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.725684 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.727528 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.728340 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.733070 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.739897 17701 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:21.740026 17701 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:06:21.740415 17701 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:06:21.740904 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:21.760134 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240503 02:06:21.761098 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:21.764315 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:21.764425 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:21.764596 17701 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:21.880839 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:06:21.938297 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.938588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:21.939319 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:21.939807 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:21.940825 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:21.962754 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.068599 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:22.069360 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.070967 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.087213 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.088961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.100090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.100926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.101593 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.102999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.103451 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.114702 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:22.115805 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:22.137233 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:22.179651 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 02:06:22.379201 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:06:22.441555 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.444007 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.447019 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.448834 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.460201 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.460839 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:22.468011 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:06:22.503966 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.511960 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.513324 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.513729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.521930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:22.542835 12619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 24 similar messages]
I20240503 02:06:22.544895 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:22.594414 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:06:22.622432 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.626762 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.628451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.635028 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.640264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:22.642210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:22.722929 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 02:06:22.784080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.786077 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.796707 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.797559 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:22.805471 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:22.821615 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:06:22.822062 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:22.937101 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.937990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.943578 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:22.943578 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:22.953558 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:22.961328 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.044826 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.046548 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.052692 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.053073 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.053198 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.060251 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:23.067739 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:06:23.156809 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.159262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.160799 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.161167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.185128 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.186605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.366868 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.368162 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.368232 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.368664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.385564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.387118 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.494927 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.502265 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.506834 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.508230 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.509858 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.511068 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.512989 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.516103 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.526949 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.534960 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.542658 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.557350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:23.665858 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 02:06:23.881832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.884524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.885702 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.886469 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.888689 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.898602 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:23.902369 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.904631 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.907928 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:23.910161 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:23.937996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:23.939147 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.027562 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.033051 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.038902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:24.043251 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 02:06:24.044812 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.046101 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.047051 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.255400 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.260552 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.262415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.262540 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.266127 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.271927 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.365507 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.366143 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.368368 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.369619 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.370863 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.390417 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.478770 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.481877 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.483960 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.484773 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.484776 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.486056 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.584707 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.586057 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.588081 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.588253 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.591246 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.596004 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:24.678622 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 02:06:24.791486 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.792871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.794919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.805414 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.807160 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.812005 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.918678 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.925525 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:24.925594 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.925689 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.927839 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:24.961139 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.967358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.967610 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:24.968835 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.969238 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:24.991236 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.008766 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:25.042791 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:06:25.066226 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:06:25.136965 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 02:06:25.287603 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:06:25.290563 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.291039 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.297684 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.299913 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.303042 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.345031 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.346700 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.356231 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.357031 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.359833 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.362344 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.380242 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.465090 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.467331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.467331 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.471901 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.482890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.500950 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.645375 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.646533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.649220 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.651856 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.652096 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.662870 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.786679 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.789314 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.790786 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.798733 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.801447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.826582 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.905884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:25.910826 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.911911 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.912585 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:25.921891 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:25.936002 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:25.998407 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.003252 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.005357 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.012387 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.020478 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.034626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:26.172271 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 02:06:26.230268 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.232378 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.233059 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.235930 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.246517 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:26.258702 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.358950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.362763 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.367226 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.371866 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.374238 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.378247 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:26.390540 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.394757 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.407917 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:26.416824 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.421675 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.423017 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:26.657335 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 02:06:26.748931 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.749883 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.752630 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.753477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.760469 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.768018 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.778126 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:26.787946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.788038 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.793439 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.797497 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.806902 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:26.827184 19285 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:26.827344 19285 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:26.827823 19285 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:26.828684 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:26.828773 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:26.828822 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:26.828958 19285 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:26.906080 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.906554 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:26.910203 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.916553 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:26.917821 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:26.934602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:26.964398 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:06:27.114809 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.116056 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.120488 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.120659 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.121176 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.139147 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.226593 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.226819 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.227532 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.229001 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.229620 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.238781 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.342036 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.343489 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.345643 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.347532 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.355902 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.370895 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:27.380261 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:06:27.389698 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 02:06:27.456996 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.459148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.462867 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.463776 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.469185 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.476414 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:27.497191 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:06:27.497922 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 02:06:27.613723 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:06:27.659942 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.663188 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.663590 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.665241 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.670385 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.680109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:27.765596 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:06:27.789899 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.794207 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.795969 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.796875 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.810424 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.813448 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:27.822655 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.824160 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.832044 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:27.835717 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:27.855706 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:27.867275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.164319 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.165009 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.171916 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.178050 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.183990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.192322 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.217088 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.223266 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.224032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.230064 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.240366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.241091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.338691 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.340462 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.342056 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.348762 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.349287 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.349805 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.495069 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.495405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.497570 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.505182 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.507056 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.508759 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:28.568718 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:06:28.649307 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.649310 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.652966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.654270 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.654652 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.668033 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.686486 19285 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:28.686643 19285 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:28.687075 19285 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:28.687899 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:28.688288 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:28.688364 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:28.688508 19285 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:28.722378 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 02:06:28.761292 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.762238 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.763610 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.768543 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.771559 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.779654 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.852638 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.853699 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:28.854398 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:28.854782 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:28.855233 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:28.864521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:29.069634 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:06:29.070500 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:29.111910 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:29.114154 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 02:06:29.118480 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.120748 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.121335 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.121726 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.123354 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.214859 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.215171 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.218798 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.228291 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.245131 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.248119 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.292361 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.296484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.296525 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.297339 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.303963 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.305016 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.548151 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.552570 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.553583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.555505 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.570173 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.592939 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.641795 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.643841 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.648180 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.658759 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.667918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.668551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.762635 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.763671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.765870 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:29.768399 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 02:06:29.769208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.769449 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.778796 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.888435 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.889585 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:29.894436 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:29.902197 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:29.908102 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:29.927865 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:29.959453 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 02:06:30.052124 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:06:30.083294 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.085561 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.087643 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.092461 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.106298 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.117167 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:30.203303 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.205143 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.220747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.221958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.222623 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.227926 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:30.262058 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:06:30.276824 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.277004 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.282301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.282982 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.289796 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:30.306464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:30.343679 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:06:30.346299 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:30.550092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.551277 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.552289 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.558271 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:30.559398 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.560155 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.648785 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.649422 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.649678 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.654785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:30.663581 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.665899 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.718186 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.724206 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.725180 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:30.725458 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:30.727335 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.728868 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:30.922853 22366 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:30.923002 22366 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:30.923456 22366 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:30.924463 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:30.924751 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:30.924825 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:30.924958 22366 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:30.989506 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.992046 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:30.999581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.003428 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.012293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.022827 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.088027 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.090942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.100051 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.101831 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.116600 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.120007 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.200074 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.201838 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.204916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.205950 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.206918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.208056 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:31.238287 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 02:06:31.320008 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.322369 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.323376 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.324018 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.325682 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.346853 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.516410 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.516842 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.521982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.522881 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.523310 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.544575 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.644954 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.649394 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:31.650399 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 02:06:31.659019 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.662885 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:31.664357 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.669399 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.670740 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.678436 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.679612 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:31.691627 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:31.696493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:31.718451 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:31.988224 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:06:32.024868 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.027035 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.028947 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.035442 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.037315 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.042209 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.057272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.069741 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.073096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.075397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:32.082283 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.093106 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:32.186707 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.190968 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.191673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.192559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.201625 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:32.221649 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 02:06:32.223695 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:32.380753 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.383654 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.384467 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.387951 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.391376 21314 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:32.391505 21314 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:32.392092 21314 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:06:32.393235 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:06:32.393507 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:32.393564 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:32.393810 21314 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:32.395341 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.407460 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:32.463702 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:06:32.508610 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.511452 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.513392 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.514827 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.517359 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.530318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:32.539234 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 02:06:32.616911 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.617616 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.618500 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.618779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.620937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.644176 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:32.709303 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.714725 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.720247 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.726576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:32.728786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.736825 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:32.826644 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:06:32.899644 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:06:32.932560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:32.932668 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.935107 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.941872 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:32.946137 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:32.961544 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:33.003155 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:06:33.056464 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.067510 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.068851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.070127 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.081259 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.082125 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:33.117009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.123034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.127306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.132838 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.136118 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.144162 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:33.464759 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.465952 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.469797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.470641 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.492841 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:33.493793 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.508914 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.515937 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.519222 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.535343 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.539459 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.550189 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:33.632764 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.637838 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.637876 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:33.639679 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.650812 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.651903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.837095 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.838367 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.849748 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.853109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.855480 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.863055 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:33.883865 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 02:06:33.934823 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:06:33.951860 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.955533 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.956990 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:33.958406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:33.966637 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:33.978085 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.061661 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.061975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.065387 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.068176 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.078985 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.105198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:34.126549 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 02:06:34.165839 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.168666 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.170503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.172453 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.187826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.191258 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.395011 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.397847 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.401885 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.401892 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.404008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.404335 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.518711 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.528045 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.530865 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.532155 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:34.534346 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 02:06:34.534958 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.561504 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.561548 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.562606 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.567958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.570032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.582674 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.593705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.699221 22366 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:34.699355 22366 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:34.699827 22366 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:34.700724 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:34.700999 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:34.701072 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:34.701253 22366 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:34.795781 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 02:06:34.857831 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:06:34.902729 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.905977 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.916072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.920679 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:34.929034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.936106 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.936661 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.941226 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:34.948112 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:34.951604 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.955461 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:34.989423 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.067945 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.070686 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.071782 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.077885 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.078120 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.079581 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:35.228163 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:06:35.272756 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.275265 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.276679 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.278728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.279898 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.301095 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.386495 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.387749 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.389576 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.391145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.399250 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.401520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.497510 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.497638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.498682 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.502944 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.508054 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.509171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:35.572901 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:06:35.614557 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.614924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.617828 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.620283 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.624245 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.644400 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.819581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.821033 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.822932 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.826435 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.834928 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.856755 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.941977 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.951242 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.951634 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.952361 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.966603 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.973767 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.982524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:35.985805 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:35.990892 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:35.991186 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:35.997561 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.004609 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.327679 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.331547 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.332751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.332908 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.336344 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.341842 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.345804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:36.355922 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.371659 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.373775 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.374889 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.397704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:36.420346 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 02:06:36.486783 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.489185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:36.490204 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.494765 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.502905 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.517658 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:36.519451 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240503 02:06:36.690593 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.691151 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.696017 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.698454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:36.706108 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.718137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.805680 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.807250 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.815135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:36.819307 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.838099 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.841269 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.931746 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.933804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:36.933881 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:36.934602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:36.942318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:36.962723 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.030591 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.031847 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.034333 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:37.034886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.035197 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.042332 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:37.074903 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:06:37.075517 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:06:37.139634 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240503 02:06:37.252705 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.256887 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.262568 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.262840 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:37.263092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.280246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.375342 22222 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:37.375476 22222 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:06:37.376020 22222 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:06:37.376684 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:06:37.383414 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:37.383579 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.385579 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.386168 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.387233 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:37.388507 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:37.388581 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:37.388762 22222 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:37.395031 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.404558 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.415663 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.420218 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.431944 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:37.432237 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.451470 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.451720 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:37.500896 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240503 02:06:37.759737 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.761176 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.764561 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:37.767956 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.779911 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.785959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.788101 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:37.804527 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.812443 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.814527 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.814637 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.836009 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:37.913973 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.914605 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:37.915449 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.926615 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:37.927774 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:06:37.928625 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:37.940356 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:37.941322 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:06:37.990224 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:06:38.122805 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.124285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.126252 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.133100 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.133880 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.141897 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.237946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.238049 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.240353 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.245173 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.252218 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.256569 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.350930 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.351292 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.351331 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.351707 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.363322 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.364389 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.453356 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.454771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.455687 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.457213 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.467512 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.480897 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.666440 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.671454 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.672209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.672859 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.673260 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.695003 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.784869 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.787214 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.794975 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.797386 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.805572 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:38.814767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.814960 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.818328 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:38.841516 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:38.850971 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 02:06:38.851900 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:38.853823 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:38.854053 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:38.927552 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 02:06:39.153903 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.154016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.156692 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.162350 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:39.163707 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:06:39.176800 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.178779 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.216323 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.217201 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.227218 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.244856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.246389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.248046 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.334491 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.335827 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.338213 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.338654 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.341289 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.352404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:39.441098 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:06:39.486833 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.491055 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.491310 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.492452 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.492560 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.513713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:39.531402 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 02:06:39.651460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.651502 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.653570 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.655093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.664959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.676188 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.763270 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.763918 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.768563 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.768767 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.769615 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.784996 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.851349 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:39.853965 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:39.856171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:39.856676 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:39.858824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:39.871699 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 02:06:39.879812 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.088227 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.091796 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.093811 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.100647 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.108525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.112072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:40.171815 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:06:40.216084 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.216873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.219789 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.221552 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.229882 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.233456 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.236761 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.238129 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.239694 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.240804 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.265867 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.266309 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:40.387575 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:06:40.589730 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.591846 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.592599 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.596649 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.599402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.603896 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.614940 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.622104 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.638919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.654021 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.654269 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.664291 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.735883 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.736894 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.737299 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.740417 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.745884 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.747372 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.957616 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:40.960389 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.961659 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:40.969367 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:40.970502 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:40.986837 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.068197 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.070076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.071425 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.072329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.079931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:41.097952 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 02:06:41.100958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.180375 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.180945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.184260 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:41.185222 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.188496 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.193430 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.288389 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.289186 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.295897 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:41.297890 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.308284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.308737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.339351 24985 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:41.339485 24985 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:41.339973 24985 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:41.340843 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:41.341696 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:41.341766 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:41.341981 24985 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:06:41.406560 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 02:06:41.496568 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.499519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.500275 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.500452 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.509615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:41.526160 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.616389 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.627645 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.631858 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.633599 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.636358 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.639925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.650312 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:41.656630 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:06:41.658208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:41.660621 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:41.677559 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.682804 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:41.692214 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:41.964234 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 02:06:41.988014 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:41.994591 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:41.995090 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.002780 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.013957 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.021739 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.026593 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.040709 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.053359 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.056751 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.057547 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.067787 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.153487 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.155501 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.157402 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.159068 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.172151 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.174268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:42.252447 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:06:42.345185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.345206 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.347626 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.349025 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.349227 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.351933 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:42.462484 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:06:42.467948 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.470997 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.472636 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.475101 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.478288 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.479578 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:42.525992 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 02:06:42.575086 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.578483 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.579524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.579857 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.584110 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.597079 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.677706 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.678050 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.685200 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.688200 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.691718 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.707544 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:42.751022 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:06:42.887493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.892403 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:42.896905 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:42.899339 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:42.903903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:42.906330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.010473 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:43.011926 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:06:43.020079 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.026101 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.028198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.028784 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.028903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.031348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.041448 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.042762 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.052991 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.062963 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.066398 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.381422 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.386780 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.396092 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.405493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.418259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.423319 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.427275 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.435523 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.447882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.457705 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.465677 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.470422 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.560483 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.565295 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.565667 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.566643 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.569512 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.594548 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.761731 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.764241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.764279 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.771057 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.777627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.790709 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:43.849511 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 02:06:43.883782 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.883996 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.889060 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:43.891232 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:43.891525 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:43.901161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:43.991451 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:43.994009 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 02:06:43.994635 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.000006 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.003463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.010572 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.018890 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:44.111572 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.116272 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.120651 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.122282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.124176 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.132098 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:44.315155 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.316861 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.316967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.317993 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.319890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.338976 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:44.351671 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:06:44.450774 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.454376 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.456749 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.465184 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.473330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.481699 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:44.492131 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.502954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.514065 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.514246 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.516121 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:44.525903 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:44.529844 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 02:06:44.573177 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:06:44.838938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.839375 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.846431 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.848029 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:44.849982 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.862250 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.889338 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.895391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.903687 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.904994 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.906162 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:44.914710 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:44.964524 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 02:06:44.989573 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.990239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:44.991101 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:44.993230 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.017421 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.020783 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:45.132565 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:06:45.220525 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.223747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.225978 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.227097 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.242486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.244393 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:45.287647 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:06:45.336716 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.338729 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.340618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.342833 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.344974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.347894 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.452387 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.454346 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.459982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:45.460213 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240503 02:06:45.460425 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.461125 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.463034 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:45.487542 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:45.558426 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.558717 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.558920 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.562646 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.564795 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.574926 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.772310 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.772783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.775592 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.776605 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.787200 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.788823 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.899896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.900820 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.901401 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.911847 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.913321 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:45.926343 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.937579 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.957322 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.958112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:45.958541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:45.960031 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:45.967918 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.289399 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.289885 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.291893 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.293294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.300830 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:46.311537 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 02:06:46.314195 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.321275 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.328487 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:46.339891 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:46.343325 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.353464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.366972 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.455276 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.459969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.462036 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:46.462438 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.464699 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.476722 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:46.554419 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 02:06:46.646296 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.646380 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.652309 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:46.660753 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.661782 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.664211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:46.761771 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:06:46.782227 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.782698 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.784061 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.784204 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.785888 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.787024 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:46.853334 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:06:46.893613 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:46.894199 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.895970 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.898674 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:46.905611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.905922 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:46.996742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:46.998805 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.001286 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:47.002074 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.017990 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.020170 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:47.117523 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 02:06:47.211825 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.212097 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.212528 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.212971 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.228214 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:47.228698 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:47.271966 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:06:47.329530 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.335927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.337379 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.341739 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.349001 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:47.351959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.369091 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.371834 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.382130 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:47.389508 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 02:06:47.390358 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.401221 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.405368 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:47.467638 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:06:47.714330 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.722460 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.726517 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.728410 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.736074 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.749405 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.756616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.764083 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.765077 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.765990 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.766475 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:47.782521 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:47.893934 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:47.895499 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.896930 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.897498 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:47.898422 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:47.899932 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:48.028250 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:06:48.028970 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:06:48.086010 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.086658 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.088068 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.097239 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.105515 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.108034 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.202054 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.204967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.207067 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.213488 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.214828 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.228129 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.317198 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.317406 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.318281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.319659 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.325057 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.328016 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.423454 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.423866 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.426486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.426546 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.443272 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.450951 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.638222 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.638705 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.638963 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.641486 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.655110 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.659529 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:48.733295 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 02:06:48.755409 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.757297 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.758558 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.760697 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.766119 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.768754 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.772055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.785096 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.805347 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:48.806977 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:48.828730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:48.829231 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:48.924903 26729 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:48.925084 26729 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:06:48.925482 26729 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:06:48.926077 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:06:48.926877 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:48.926968 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:48.927049 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:48.927196 26729 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:49.114300 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.117046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.119742 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.130435 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.148051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.150179 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:49.171293 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.185189 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.191061 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.191874 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.198673 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.211889 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:49.224593 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 02:06:49.291549 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 02:06:49.295342 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.297614 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.300808 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.309254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.312809 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.329886 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:49.389801 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:06:49.457351 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.458283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.462888 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.465039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.470079 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.474262 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:49.613788 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.614362 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.619581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.619705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.620009 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:49.630208 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.720032 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.723062 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.723155 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.726651 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.730100 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:49.744323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.821596 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.822391 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:49.824113 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:49.829655 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:49.836141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:49.836753 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:49.904258 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 02:06:50.034324 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.034574 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.036659 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:50.037979 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:06:50.038861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:50.045377 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:06:50.049300 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.059568 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.161290 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.161382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.163515 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.170884 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:50.180245 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.180662 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.181134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.192703 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.194275 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.203336 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:50.203430 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.208063 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.326390 24985 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:06:50.326546 24985 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:06:50.327113 24985 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:06:50.327940 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:06:50.328145 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:50.328222 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:06:50.328338 24985 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:50.525346 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.526907 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.527459 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.530853 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:50.540295 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.546298 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.577683 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.578253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.579360 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.584800 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.593577 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:50.600881 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:06:50.610087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.691892 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.693274 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.693449 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.694087 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:50.696410 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.725586 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.890329 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.893373 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.895418 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:50.903189 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:50.903537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:50.916955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.015270 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.019135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.022724 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.028498 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.029153 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.049047 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.127633 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.132876 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.133003 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.135280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.139168 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:51.139676 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240503 02:06:51.165287 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.228477 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.228920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.233183 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.244303 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.246484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.247242 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.269675 26594 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:51.269804 26594 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:06:51.270259 26594 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:06:51.271422 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:51.271509 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:51.271591 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:51.271709 26594 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:51.469312 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.469545 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.471434 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.471532 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.473667 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.497112 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.580463 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.587482 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.589915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.600488 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:51.606706 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 02:06:51.613636 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 02:06:51.617558 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.620452 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.621076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.628082 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.630087 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.638576 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.643931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.655540 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.968658 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.969203 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.970043 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.973872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:51.977450 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:51.987771 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:51.993157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:51.998145 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.001766 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.009914 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.018015 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.035821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:52.091205 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240503 02:06:52.123720 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:52.124893 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:06:52.125238 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.126721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.127400 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.136409 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.155671 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:52.325820 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.326484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.326915 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.330818 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:52.337774 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.348865 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.441227 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.442703 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.442883 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.444443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.462649 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.475908 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:52.539848 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:06:52.552364 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.552959 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.559739 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:52.564303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.564585 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.564818 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:52.606617 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:06:52.645161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.645169 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.649734 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.651607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.658905 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.669965 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:52.872628 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.876958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.878602 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:52.880744 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.884346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.892304 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:52.990640 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:52.994542 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:52.994741 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.007742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.008415 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.009615 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.012993 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.015925 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.015941 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.036042 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.043627 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:53.068619 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:06:53.069434 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:06:53.076012 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.363821 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.365052 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.370994 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.375587 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.383936 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.386487 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.404780 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.413928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.419044 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.428665 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.443276 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.445600 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.535346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.537087 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.542362 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.545035 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.545729 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.557061 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.730561 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.735527 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.736028 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.736320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.747380 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.751057 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:53.857327 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.860265 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.863730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.867933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.868264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:53.869314 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 02:06:53.877979 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.963887 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:53.966017 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.966084 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.970594 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:53.973245 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:53.984767 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.011902 28313 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:06:54.012022 28313 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:06:54.012459 28313 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:06:54.013175 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:06:54.014214 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:06:54.014602 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:06:54.014693 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:06:54.015053 28313 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:06:54.074064 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.076138 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.080147 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.080696 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.081238 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.081899 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:54.170537 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240503 02:06:54.286868 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.287272 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.297366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.297502 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.298936 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.312885 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:54.412478 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.414353 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.416409 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.416965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.417013 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.419656 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.438951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:54.440210 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 02:06:54.451153 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.454538 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:54.463528 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.492879 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:54.494007 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:54.654511 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 02:06:54.737195 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:06:54.775538 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.777405 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.781757 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.791533 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.794214 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.821647 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:54.831154 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.846946 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.853449 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.863281 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.870836 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.871246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:54.896143 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:06:54.944386 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.944566 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.950305 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:54.956212 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:54.958662 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:54.977219 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.166075 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.166847 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.170564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.179637 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.180246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.182462 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.273679 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.276087 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.276696 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.277673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.290593 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.301795 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:55.335605 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:06:55.383369 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.384574 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.389626 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.391280 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.395911 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.396217 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:55.459501 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:06:55.491150 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.494848 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.495890 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.496517 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.497566 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.498387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.716333 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.716778 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.721369 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.721779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.726018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.738337 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.836160 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.844623 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.854430 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.855458 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.866784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.873456 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.874246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:55.874617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:55.876246 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.890347 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:55.891554 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:55.916595 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.216547 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.225263 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.225641 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.226132 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.226791 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.231913 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.237977 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:56.244482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:56.260076 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240503 02:06:56.261122 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:56.266224 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.277830 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.300123 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.366619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.367882 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.374423 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:56.378484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.384305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.387436 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:56.564743 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240503 02:06:56.581935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.583109 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.590615 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.596693 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.602241 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:56.603917 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.690579 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.694373 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.695924 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.696159 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.696764 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.723513 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:56.797312 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.799237 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.809840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.811094 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.811450 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.813812 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:56.884195 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240503 02:06:56.909937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:56.913113 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.914038 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.914391 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:56.916850 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:56.931816 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:57.130204 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.130671 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.133925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.139437 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.148085 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.148380 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:57.167368 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:06:57.181310 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:06:57.201125 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240503 02:06:57.238978 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.249265 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.257272 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.263458 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.267596 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:57.282284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.301853 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.303799 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.305477 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.317169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.318603 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.320856 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:57.631116 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.639910 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.642594 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.642808 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.645275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.647164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.651861 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.666836 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:57.667184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.675225 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.695739 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.701308 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:57.758558 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:06:57.781852 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.782115 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.788771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:57.790160 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.798523 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.815461 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:57.848270 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:06:57.986701 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.988895 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.991514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:57.992048 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:57.997406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.026530 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.114481 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.116950 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.117697 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.121227 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.121353 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.138892 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.220162 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.224800 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.226758 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.231155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.239358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.247923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:58.257791 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:06:58.329044 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.330175 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.331384 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.333407 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.335436 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.343245 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.538156 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.540581 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.542709 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.543856 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.544003 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.547953 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.660532 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.665633 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:58.669553 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.677959 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.681046 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.681293 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.684353 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:58.688056 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.688212 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.711671 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:58.722322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:58.733654 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:06:58.830798 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 02:06:58.982578 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 02:06:59.034866 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.036108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.040470 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.048463 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.049187 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:59.059569 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.066000 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:59.066330 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.078377 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.092093 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.094645 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.124070 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.204730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.204839 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.212241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.212343 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.212605 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.221906 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:59.388960 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:06:59.398389 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.398720 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.402458 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:59.405110 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.416690 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:06:59.422859 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:06:59.425834 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.517987 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.520530 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.521900 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:59.531513 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.532323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.554965 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.633396 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.633481 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.634459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.635022 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:06:59.635560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.638345 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:06:59.656659 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240503 02:06:59.735886 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.738544 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.739992 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.742909 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.753060 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.756589 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:06:59.831127 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240503 02:06:59.951581 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.951887 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.954962 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:06:59.959928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:06:59.974298 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:06:59.980331 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.067211 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.073132 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.073557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.074460 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.088099 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.104346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.136848 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.137760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.139454 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.140980 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.148313 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.152570 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:00.277087 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:07:00.402277 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.407688 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.409157 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.409389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.420459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.426623 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.496501 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.496600 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.498253 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.498677 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.498957 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.514647 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:00.539213 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:07:00.598258 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.599969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.599969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.602277 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.603638 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.617424 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.733186 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.734285 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.735963 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.739568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.745018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.746898 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.903160 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.904074 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:00.912724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:00.912807 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:00.913123 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:00.924432 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.022987 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.027842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.029879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.040220 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.040498 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.072283 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:01.094048 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.095394 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.101656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.108156 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.113883 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:01.116540 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.423079 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.427354 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.427397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.430366 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.433355 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.448943 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.449635 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.451557 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:01.460074 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.463349 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:01.464161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.471844 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.574584 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.579716 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.579856 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.581431 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:01.586560 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240503 02:07:01.590675 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:01.596995 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 02:07:01.597131 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.789603 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.789872 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.790796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.797483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.816282 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:01.822456 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:01.837808 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:07:01.895365 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.896986 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.897521 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:01.897917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:01.898806 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:01.915930 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:01.960364 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:07:02.003404 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.003829 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.012481 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.013671 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.014597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.021556 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:02.053997 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 02:07:02.119874 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.120437 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.122455 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.130625 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.142467 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.144075 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.310081 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.313455 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.317075 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.323182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.331903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.338163 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.440868 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.456876 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.457325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.459987 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.471601 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.480144 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.491046 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.492225 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.499372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.499478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.504185 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.512548 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:02.577391 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 02:07:02.623374 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:07:02.832675 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.833143 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.834403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.840497 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.846048 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:02.854696 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:07:02.860105 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.861194 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.867969 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.876009 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:02.877475 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.896394 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.900835 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.975447 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.975745 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.981488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:02.986507 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:02.994787 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:02.998332 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.191476 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.195235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.196660 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.198455 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.203318 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.203408 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.297560 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.298369 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.302024 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.305713 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.314674 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.328394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.401818 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.408478 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.409554 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.412293 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.415196 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.438169 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.515018 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.519802 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.519928 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.520920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.531948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.532886 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:03.594934 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:03.724788 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.724828 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.725482 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.730891 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.741681 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.754113 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.852159 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.856971 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.857439 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.860769 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.871372 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.880535 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.891664 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.894161 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:03.898329 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:03.905282 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:03.913635 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:03.930114 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:04.044008 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 02:07:04.081321 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:07:04.135527 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 02:07:04.226552 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.238945 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.239625 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.242533 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.244619 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.245265 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.245873 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.249576 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.260052 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.284929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.286360 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.293983 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:04.374451 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:07:04.383787 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.385663 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.389993 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.390394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.402208 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.405856 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.587436 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.588685 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.589867 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.592486 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.596761 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.613991 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:04.646622 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240503 02:07:04.700007 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.700282 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.703087 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.704385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.706857 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.707896 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.803769 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.803776 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.803975 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.814532 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.820657 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.831923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.911571 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:04.911883 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.914368 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:04.917145 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:04.919481 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:04.932126 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.123442 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.124013 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.124303 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.126668 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.127200 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.143580 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:05.145439 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:07:05.232827 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.240615 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.243861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.245270 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:05.252409 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.258757 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.271157 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.273677 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.274518 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.289618 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.297853 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.313093 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:05.419224 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240503 02:07:05.596518 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.598695 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.602882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.603420 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.606676 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.610641 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:05.661972 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.665556 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.667078 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:05.677222 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:07:05.678068 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.679234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:05.685621 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.756950 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.757829 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.761857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.770977 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.772169 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:05.787814 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.986989 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.987169 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:05.990962 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:05.991222 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:05.993592 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:05.993904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.085029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:06.085126 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.088863 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:06.093503 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.104316 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.110184 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.186967 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.187608 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:06.190920 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.198854 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.200004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:06.217293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.298053 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.298110 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.302835 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.307559 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.307914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:06.310998 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:06.338861 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 02:07:06.501986 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.502140 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.505718 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.505923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.511907 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:06.522089 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:06.569386 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:07:06.611907 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:06.621281 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.624468 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.634059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.634285 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:06.636274 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.638307 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.652230 28313 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:06.652329 28313 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:06.652694 28313 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:06.658706 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240503 02:07:06.658872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:06.662868 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:06.663130 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:06.663151 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.663189 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:06.663311 28313 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:06.667623 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:06.675936 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:07:06.683094 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.691730 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:06.785395 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 02:07:06.988036 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.989928 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:06.997287 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:06.999905 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.003726 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.008903 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.013072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.015928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.025625 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.032243 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.044260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.056998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.138163 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.141531 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.143631 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.144913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.155913 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.157630 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:07.304658 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 02:07:07.324564 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:07:07.359555 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.360985 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.361604 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.361824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.362644 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.363644 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.461549 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.464216 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.465472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.470484 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.480856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.487541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.564435 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.564813 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.567198 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.572110 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.596323 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.598002 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.671669 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.671728 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.673195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.686725 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:07.686856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.700767 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.889302 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:07.891172 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:07:07.892745 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.893043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:07.893289 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:07.906030 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:07.906430 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:07.913261 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 02:07:08.011497 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.011592 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.021481 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.021773 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.030440 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.054395 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.062404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.067898 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.069474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.070045 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.071326 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.086319 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.371882 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.372989 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.381898 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.383718 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.385772 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.386093 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.450484 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.457351 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.457618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.459367 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.466310 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.476045 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.501540 32107 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:08.501655 32107 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:08.502220 32107 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:08.502833 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:08.503122 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:08.503196 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:08.503332 32107 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:08.555169 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.556859 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.561733 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.562553 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.572022 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.579862 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.688478 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.693410 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.696079 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.698848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.705112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.714700 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:08.725121 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:07:08.804777 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 02:07:08.869943 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.871873 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.872296 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.876384 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.882529 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.892601 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:08.974022 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.975389 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.975664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:08.986163 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:08.988809 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:08.992259 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.047894 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.052094 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.053635 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.056924 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.064177 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:09.074088 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:07:09.082726 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:09.088894 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:07:09.290609 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.291910 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.294641 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.294721 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.294952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:09.309474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.388970 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.390370 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.395354 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.421425 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.423316 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:09.427479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.456734 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.458982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:09.464144 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.464394 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.466629 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.487349 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:09.500643 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240503 02:07:09.734606 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:09.736132 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:07:09.739123 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.739948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.746456 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.762935 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.772225 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.779282 1105 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:09.779394 1105 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:07:09.779875 1105 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:07:09.781751 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:09.781951 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:09.782017 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:09.782342 1105 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:09.825664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:09.826273 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.827930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.829792 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.835922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.846419 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:09.930192 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.930794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:09.931187 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:09.937645 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:09.938992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:09.948856 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 02:07:09.956959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.059113 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.059401 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.060413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.065074 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.070111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.083981 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:10.159240 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 02:07:10.238760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.241235 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.246793 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.250633 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.253486 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.264389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.349061 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.353765 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.354012 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.355894 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.358278 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.359963 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.428804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.431979 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.432854 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.441694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.456208 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.463861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.660712 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.663091 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.664654 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.665053 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.672770 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.680294 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.774940 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.775154 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.777838 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.779700 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.801050 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:10.804262 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.808126 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:10.812173 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.812202 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.821908 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:10.839306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:10.847118 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:10.947088 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:07:11.089823 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240503 02:07:11.151082 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.158788 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.162273 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.162387 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.166625 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.172003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.172988 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.174353 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.208523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.208860 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.211454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.213845 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.314257 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.316120 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.316751 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.316792 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.318779 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.341351 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.515399 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.516814 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.519191 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.523370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.524257 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:11.535460 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:07:11.538060 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:07:11.543320 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.625366 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.629217 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.629913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.630888 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.632841 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.640503 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.729185 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.731842 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.736884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.744489 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.744851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.745028 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.838686 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.839548 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:11.841240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:11.844134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:11.858120 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:11.868669 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:11.969164 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 02:07:12.037845 1732 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:12.037981 1732 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:12.038341 1732 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:12.039994 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240503 02:07:12.044626 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:12.045761 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:07:12.045853 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:12.045931 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:12.046152 1732 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:12.048377 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.048524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.051345 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.054597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.078737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:12.103435 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:07:12.182384 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.188398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.191262 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.202602 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.204286 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.212517 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.233402 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.233954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.239917 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:12.250643 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.252842 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:12.264686 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.402220 2235 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:12.402356 2235 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:07:12.402750 2235 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:07:12.403720 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:07:12.403834 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:12.403914 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:12.404054 2235 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:07:12.423136 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 02:07:12.502374 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240503 02:07:12.560810 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.567121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.567548 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.570103 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.570837 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:12.579483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.609043 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.616436 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.618878 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.630637 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:12.633863 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.643433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.716548 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.718549 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.719436 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.722903 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:12.728247 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.745616 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.928471 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.931906 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.935114 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:12.935169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:12.936923 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:12.962924 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:12.988571 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:07:13.039685 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.040166 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.040755 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.042896 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.043308 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.053956 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.138149 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.142179 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.144564 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.145931 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.148830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.153025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.250447 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.250775 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.262445 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.263743 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.275295 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.278692 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.456151 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.461627 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.463527 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.463982 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.468669 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.481668 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.580756 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.590767 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.591078 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.591372 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.595350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.598135 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:13.604362 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.628737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.632179 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.641161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.648319 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:13.650676 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:13.669813 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 02:07:13.755046 1732 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:13.755175 1732 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:13.755558 1732 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:13.756639 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:07:13.756783 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:13.756865 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:13.756990 1732 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:13.961762 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.962738 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.967406 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.968333 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:13.969393 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.974783 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:13.981377 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:13.983652 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:13.993947 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.010952 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.013901 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.019987 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.038720 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:14.079459 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:07:14.114861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.116644 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.125736 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.125967 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.134135 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.144032 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:14.174054 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:07:14.328461 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.329085 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.331291 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.333107 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.335582 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.347772 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:14.405718 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 02:07:14.439827 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.440261 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.440704 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.450791 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.451485 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.469105 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.549261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.549911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.551177 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.551327 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.556605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.560896 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:14.569303 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:07:14.647540 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.648295 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.652397 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.656997 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.657716 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.658071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.872687 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.873281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.875360 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.876475 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.888060 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.889808 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.975862 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.976251 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:14.982136 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:14.985268 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:14.990069 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:14.998838 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.045641 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.046443 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.050859 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.055087 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.058354 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.076699 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:15.110621 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 02:07:15.144080 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 02:07:15.297683 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.301613 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.304505 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.306161 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.314401 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.324889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.387365 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.389017 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.392817 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.395876 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.405150 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.412853 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.507102 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.508617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.511529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.517866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.518568 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.527287 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.613401 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.613600 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.618168 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.620177 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.620991 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.634562 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.837152 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.839699 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.843254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:15.852254 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:15.870281 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.875274 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.964212 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.965493 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.966854 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:15.969367 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:07:15.980669 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:15.986295 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:15.996979 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:16.017129 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 02:07:16.028704 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.031322 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.035830 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.038224 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.050580 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.052340 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:16.294560 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.295055 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.310695 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:16.321477 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.322122 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.327359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.383996 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.385767 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:16.388741 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.389607 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.392318 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.418848 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.494107 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.496836 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.497756 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.498275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.512215 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.516063 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:16.524000 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:07:16.608359 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 02:07:16.613775 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.615923 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.617130 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.617453 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.621290 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.636871 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:16.806329 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.809510 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.811391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:16.821323 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:07:16.823352 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.833278 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.838609 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:16.841573 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:07:16.940130 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.941192 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.946746 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.946964 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.950685 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:16.950716 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.966646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:16.973309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.987876 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:16.990271 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:16.999862 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.014729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.308339 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.316442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.317406 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.318938 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.327426 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:17.337651 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.354183 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.355932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.356168 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.361196 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.379974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:17.395498 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.476116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.477557 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.479821 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:17.481029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.490499 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.500371 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.644953 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.645522 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.648245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.660677 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.661918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.662741 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:17.699321 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 02:07:17.788516 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.790516 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.790859 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.791317 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.795632 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:17.812644 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:17.884917 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 02:07:17.908649 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.911207 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.912459 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:17.920384 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.932407 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:17.943405 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:17.997421 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:17.997749 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.003365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:18.005053 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.016085 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.031440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:18.105654 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:07:18.233553 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.234817 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:18.234916 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.247210 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.249493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.252357 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:18.285351 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:07:18.289304 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:18.347118 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.356588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:18.357507 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.358000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.364264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.384951 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.386613 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.393996 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.398106 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.407717 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.413287 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.419636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:18.717764 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 02:07:18.725056 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.726658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.729589 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.734508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:18.740602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.748680 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.798687 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:18.800004 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.804697 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.807356 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.810186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.844803 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.899737 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.902523 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.904989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:18.908607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:18.922307 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:18.936254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:18.985204 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:07:19.015233 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 02:07:19.048597 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:07:19.132356 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.139057 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.140189 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.143119 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.147913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.162370 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:19.239854 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.239962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:19.241951 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.261111 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.261847 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.261853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.341609 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.341770 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.351492 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.353534 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.374907 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.375412 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:19.394644 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:07:19.459906 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.463201 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.468009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.469071 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.471138 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.485667 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:19.535982 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:07:19.565816 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:07:19.659631 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.660037 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.668061 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:19.670605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.672602 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.677762 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.774400 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.777123 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.785737 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:19.788869 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.805064 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.806957 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:19.810763 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.821979 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:19.822968 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.825606 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:19.827657 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:19.863916 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:20.083017 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 02:07:20.153172 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.163746 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.163870 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.167655 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.172009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.195595 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.204587 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.209547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.218609 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.227499 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.230360 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.246160 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.318625 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.321185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.322037 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.327886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.328166 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.336959 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.536976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:20.537307 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 02:07:20.540764 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.544503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.545289 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.545972 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.546960 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.641435 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.645354 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.645900 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.646212 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.646585 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.663851 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.745097 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.746969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.749713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.755383 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:20.758683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.765192 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.854544 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.859911 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.862939 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:20.863535 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:20.869344 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:20.877709 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:20.908043 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 02:07:20.999830 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:07:21.060586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.061251 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.061517 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.061798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.080276 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.080452 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.182154 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.183521 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.188623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.193806 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.205549 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.211097 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.219877 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.228677 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.234476 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.240625 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.257378 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.267541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:21.424964 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 02:07:21.429838 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:07:21.562741 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.565532 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.566043 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.567529 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.584724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.586414 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.596717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.599313 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.606843 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.607884 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.608764 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.635525 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.719594 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.720732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.720780 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:21.721004 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.725268 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.736300 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.916023 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.916687 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.920576 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:21.925391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:21.926954 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:21.945415 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:21.952158 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:07:22.030272 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.032547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.033054 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.033844 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.036046 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.047940 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.135018 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.136672 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.137467 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.139302 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.144009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:22.144929 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:07:22.146427 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.234247 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.237562 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.239125 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.240710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:22.245573 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240503 02:07:22.247036 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.266971 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.449280 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.450846 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.454980 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.459941 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.462451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.463388 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.570616 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.572023 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.572451 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.576227 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.582902 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.583866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.594377 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.604745 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.615586 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.619127 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.627542 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.644577 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.918865 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.919575 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.922736 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.934541 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.937577 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.944504 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.985787 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.986837 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:22.991219 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:22.992041 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:22.993299 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:22.994004 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.005684 3729 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:23.005800 3729 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:07:23.006259 3729 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:07:23.007292 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:23.007432 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:23.007478 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:23.008013 3729 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:23.090492 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.091254 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.095929 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.098822 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.099769 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:23.107146 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:23.156514 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:07:23.178690 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240503 02:07:23.236490 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.243716 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:23.247964 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.250270 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:23.252264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.261643 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.405529 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.408752 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.410667 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:23.412180 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.422798 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.424028 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:23.500332 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 02:07:23.530701 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.533828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:23.536044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.536418 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.536967 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.564410 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:23.599089 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.600965 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:23.604612 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.608011 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.612897 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.617013 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:23.716223 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 02:07:23.890656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:23.891674 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.900463 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.901261 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:23.902132 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.906314 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.967092 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.969866 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:23.974370 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:23.977866 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:23.987162 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:24.009461 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:07:24.010874 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:24.050961 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.051653 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.054208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.056177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.063938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.066368 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:24.283033 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:24.303143 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.307148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.310232 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.310254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.310667 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.343329 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:24.403502 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.405011 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.407235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.412853 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.420747 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:24.428917 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.512305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.512317 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.513547 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.525933 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.528214 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.539583 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:24.624745 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.629948 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.639967 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.641124 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.651016 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.651830 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:24.688571 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:07:24.835000 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.835484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.839010 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.846980 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.855880 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:24.870752 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:24.872779 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:07:24.886953 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 02:07:24.962756 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.965245 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.967911 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.973608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:24.975013 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.985579 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.997404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:24.998383 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.000653 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.005666 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.006939 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.019097 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.325431 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.338199 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.338641 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.343569 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.349871 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.364374 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.384332 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.397890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.400280 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.402093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.402093 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.411517 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:25.486271 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 02:07:25.506070 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.508621 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.512639 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.519867 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.522140 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.522773 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.682854 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.685989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.686272 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.689787 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.689951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.690171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.823408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.823560 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.830389 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:25.832897 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.834805 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.838089 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:25.896183 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 02:07:25.953557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:25.955595 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.956848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:25.959384 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.959967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:25.977880 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.044283 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.045717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:26.046424 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:07:26.050071 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.050451 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.050711 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.056111 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.104293 6301 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:26.104420 6301 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:26.104915 6301 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:26.105705 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:26.106108 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:26.106177 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:26.106309 6301 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:26.264709 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.266274 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.273706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.278623 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.290042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.300011 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.368265 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.369645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.371949 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.374742 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.375264 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.402755 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.441380 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.446818 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.448747 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.451738 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.456617 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.460507 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:26.478230 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:07:26.589074 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 02:07:26.689210 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.695609 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.696637 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.697419 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.697710 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.698518 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.784034 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.784801 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.792676 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.793851 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.804677 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.809064 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:26.907855 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.908663 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.912724 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:26.916699 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:26.920784 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:26.924981 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.014508 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.018512 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.019632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.028210 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.037710 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.037945 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:27.206388 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 02:07:27.223048 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.223824 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.228355 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:27.228860 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.237494 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.240025 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:27.319842 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:07:27.347497 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.351116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.358424 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:27.363826 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.364990 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.372385 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.374186 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.380321 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.381850 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:27.382282 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:07:27.390501 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.415731 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:27.433667 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:27.703729 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 02:07:27.720666 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.726667 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.730669 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.741261 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.741602 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.749737 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:27.785225 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.790156 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.796144 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.803192 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:27.808740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.813215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.892820 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.893791 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.899412 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:27.901352 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:27.909626 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:27.911991 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.105214 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.109220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.117528 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.117974 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:28.121711 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.122670 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.214659 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.215823 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.218025 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.219780 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.223909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:28.224979 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:28.226203 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.252483 1732 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:28.252616 1732 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:28.252971 1732 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:28.254101 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 02:07:28.255120 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:28.255810 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:28.255869 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:28.256034 1732 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:28.321966 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.323025 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.325264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.327922 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:28.328667 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.340308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:28.344633 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 02:07:28.428143 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.430640 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.431617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.434237 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:28.446167 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.448061 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.631451 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.633121 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.636446 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:28.637609 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.644137 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.666159 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.758772 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.760123 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.770959 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.771610 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.778843 6301 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:28.778952 6301 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:28.779369 6301 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:28.780529 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:28.783192 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:28.783277 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:28.784643 6301 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:28.791916 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:28.795845 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.804754 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:28.806164 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.808466 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:28.812543 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.817193 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:28.820396 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:29.144964 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 02:07:29.152926 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.154342 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.157241 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:29.159037 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:07:29.166227 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.167044 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:29.167141 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.172283 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.187445 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.189509 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.198345 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:29.199976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.203778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.301280 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.304334 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.307847 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.312613 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.312681 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.317584 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:29.424400 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:07:29.510452 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.511499 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.515447 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:29.518623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.519515 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.534008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.613555 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.613890 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.616247 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.618824 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.633414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.634428 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:29.715704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.717172 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.720907 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.722433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.729106 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.730894 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:29.826296 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.827178 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.827560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:29.827673 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:29.830991 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:29.851125 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:29.918726 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:07:29.978811 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 02:07:30.035602 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:07:30.036303 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.037453 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.039549 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.041129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.047938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.060175 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.151598 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.153344 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.155081 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.166529 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.169723 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.179628 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:30.183784 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 02:07:30.217501 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.219208 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.219883 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.220862 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.221841 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.222107 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.481256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.485467 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.485857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.485903 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.486793 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.488453 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.567538 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.571341 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.572680 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.573386 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.575747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.602627 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.681342 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.686674 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.691900 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.694793 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.697316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.708096 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.799834 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.803748 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:30.807474 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.824949 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:30.825090 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:30.826037 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:30.919634 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 02:07:30.995501 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.996150 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:30.996730 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.000264 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.000689 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.016968 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:31.102453 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:07:31.116569 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.121604 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.123364 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.124755 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.125272 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.134101 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.168540 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.171936 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.176268 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.188284 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.188532 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.207450 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.287055 7110 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:31.287185 7110 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:07:31.287585 7110 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:07:31.288630 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:31.288841 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:31.288936 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:31.289063 7110 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:31.495121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.495325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.496140 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.496212 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.504279 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.523787 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.551251 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.551626 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.557459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.564457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.575512 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.589987 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.647693 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.651227 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.654325 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.657544 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.663055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.669209 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:31.674294 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:07:31.694242 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 02:07:31.882972 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.884335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:31.886065 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.900785 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:31.901490 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:31.915850 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:31.997452 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.001463 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.002588 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.005015 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.019374 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.022154 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:32.108747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.109211 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.119798 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.120277 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.120328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.135393 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:32.208794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.212913 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.213759 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.219619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.224074 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.246006 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:32.267251 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:07:32.433182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.435139 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.435232 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.439476 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.445513 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.470794 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:32.522678 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 02:07:32.555027 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.559013 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.559805 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.572492 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:32.575906 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:32.579252 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:07:32.598403 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.600610 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.608829 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.611450 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.620142 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.620709 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.620944 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:32.924144 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.929350 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.934340 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.940570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.945991 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.948911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.956441 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:32.960330 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:32.960951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.972795 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:32.992728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:32.993618 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.077010 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.083981 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.085065 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.085690 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.088795 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.089277 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:33.131551 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 02:07:33.284097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.284644 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.293407 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.293829 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.295156 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.305246 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:33.320770 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:07:33.396109 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.400142 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.400563 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.404239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.405200 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.412528 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.511379 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.519389 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.520841 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.528998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.531160 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.537933 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.638168 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.638962 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.641184 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.641605 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.643733 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.645610 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:33.716862 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 02:07:33.842631 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.843128 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.849740 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.852053 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.852732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.877385 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:33.960541 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.966369 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:33.971864 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:33.979723 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:33.982615 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.002409 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.010032 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.012080 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.020710 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.022673 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.026927 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.033032 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:34.083571 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 02:07:34.166651 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:07:34.341897 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.342435 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.349433 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.349728 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.357446 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.362545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.367316 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.371896 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.372977 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.382632 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.400993 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.405361 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.506803 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.508075 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.508390 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.508406 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.510489 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.524415 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:34.568840 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:07:34.569591 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 02:07:34.634876 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:07:34.704829 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.708460 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.709123 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.717523 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.718940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.720952 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.825263 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.828446 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.828764 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.842057 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.843600 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.851570 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.944597 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.945418 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:34.947083 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:34.947474 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:34.957823 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:34.959383 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.049569 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.055948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.058889 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.059091 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.059572 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.059875 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:35.127544 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:07:35.263501 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.265389 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.270388 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.278074 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:35.279930 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.291390 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.382305 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.388551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:35.389340 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.396077 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.398519 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.409533 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.413435 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.421742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:35.422363 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.423853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.424291 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.448459 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:35.621549 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 02:07:35.771400 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.781198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.784992 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.784997 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.785856 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.791980 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.793409 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.802093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.822832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:35.823359 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.836206 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.845577 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:35.963198 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:35.964375 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.967151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.971879 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:35.972520 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:35.977277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.135388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.140421 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.142765 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:36.149065 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.150715 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.152282 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.280683 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.281324 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.285845 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.285843 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.286289 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.287281 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:36.386051 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.386847 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.388190 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.388649 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.389431 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:36.399132 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 02:07:36.399441 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:36.437709 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:07:36.438489 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:07:36.489287 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.490223 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.490334 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.494055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.512472 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.514586 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:36.643236 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 02:07:36.702143 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.706550 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.709919 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:36.716686 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.716959 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.719201 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.813978 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.817234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.818861 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:36.818900 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.828408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.830046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.848423 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.864192 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:36.867261 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:36.887166 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:36.888967 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:36.890055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:37.049958 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:07:37.190812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:37.198874 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 02:07:37.199213 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.205274 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.208762 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.209620 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.222715 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.250802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.258284 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.258543 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.266086 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.267369 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.268236 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.361160 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.367410 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.368858 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.369040 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.370754 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.378176 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.549700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.552160 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.555825 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.556232 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.558320 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.565284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.677117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.679695 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.679940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.683509 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.686589 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.692427 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.781150 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:37.783530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.784459 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.785868 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.795931 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.807555 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:37.825661 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:07:37.892185 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.892813 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.894871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:37.902896 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:37.903576 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:37.914942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:38.013398 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 02:07:38.096510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.098124 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.101104 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.104521 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.121935 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.138823 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:38.223512 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.228446 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.229552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.231942 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.237634 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.243077 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:38.245802 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.263092 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.278348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.297703 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:38.303539 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.304492 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:38.354554 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:07:38.597666 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.601583 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.601974 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.609009 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.615904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.626473 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:38.644409 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.648008 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.655179 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.660593 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.676728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:38.678123 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.757756 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.758581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.760946 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.762483 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.774739 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.786475 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:38.866722 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:07:38.886992 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 02:07:38.936453 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.941726 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.942847 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:38.944231 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:38.948046 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:38.958508 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:39.067692 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 02:07:39.073175 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.076751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.077454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:39.078779 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.089082 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.091729 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.180008 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.181747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:39.182957 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.184463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.188148 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.201840 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.279831 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.282613 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:39.283921 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.286391 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.292871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.295997 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.503711 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.505699 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.508689 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:39.513146 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.514381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.518940 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.619618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.623623 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.630086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:39.630576 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.632176 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.657624 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.670583 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.677440 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:39.679028 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:39.679415 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:07:39.680189 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.688109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.688263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:39.710912 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:07:39.711619 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 02:07:39.989985 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.990304 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:39.994540 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:39.998016 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.005472 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.005745 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.060142 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.062356 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.063120 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.065317 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.065433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.088900 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.165776 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.170464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.171617 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.174497 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.175129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.182749 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:40.280726 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 02:07:40.309260 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:07:40.329126 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.329552 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.335937 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.336395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.337061 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.358354 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.484592 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.485251 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.486935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.487318 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.487491 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.506122 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.588253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.590575 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.591895 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.593586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.594436 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.616672 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.679033 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.682026 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.686455 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.689817 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.699883 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:40.700953 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.911669 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:40.911978 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.915642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.916055 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:40.918466 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:40.931782 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.030022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.043906 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.045572 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.046700 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.051877 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.056089 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.057188 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.065444 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:41.070060 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.070159 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:41.084928 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.087904 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:41.278131 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 02:07:41.389714 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.398578 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.400130 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:41.402189 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.404717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.409962 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.447371 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.448880 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.452775 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.457584 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.461599 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.462558 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:41.544117 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 02:07:41.565392 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.568747 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.571833 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.574321 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.582482 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.583773 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:41.692092 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:07:41.692802 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:07:41.776530 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.777420 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.779632 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.779997 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.783511 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:41.788094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.886111 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:41.889957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.890672 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.892903 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:41.900233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.909291 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:41.996842 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:41.998919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.001744 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.003937 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.015573 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.023924 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.104357 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.108990 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.108986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.118822 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.119901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.133965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:42.162075 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:07:42.182932 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 02:07:42.312323 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.312618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.314836 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.316912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.319550 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.326052 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.431941 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.454919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.456887 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.458600 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.463974 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.468834 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.473548 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.486867 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.494081 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.499727 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.500308 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.500828 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:42.675940 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 02:07:42.820091 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.825906 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.826594 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.833163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.844448 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:42.854820 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.868304 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.871316 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.873726 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.876837 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.884969 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.892282 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.979339 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.983829 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:42.984453 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:42.991910 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:42.992651 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:42.992972 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:07:42.993642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.177928 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.178321 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.186703 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.187884 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.196389 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:43.203099 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.298669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.299458 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.299649 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.300199 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.307907 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:43.313802 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.400101 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.402230 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.404314 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.408255 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.408744 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:43.410676 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:43.486119 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:07:43.505805 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.507877 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.509150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.511389 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.519939 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.532086 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:43.716811 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 02:07:43.723568 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.731240 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.735543 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.735989 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.737579 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:43.738992 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 02:07:43.759045 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:43.862744 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.867836 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:43.868278 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.870298 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.874454 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.877782 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.879717 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.883212 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.912417 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:43.912611 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:43.921344 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:43.924741 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:44.213379 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:07:44.226897 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.227167 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.237911 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.238052 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.244800 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.258630 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.294610 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.296123 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.299141 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.300463 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.315259 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.322973 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.400818 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.402740 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.412043 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.412336 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.413234 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.431993 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:44.496289 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:07:44.556391 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.557762 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.562227 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.570433 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.571907 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.575165 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.726547 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.728233 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.730453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.732185 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.739953 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.743407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:44.755694 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:07:44.757681 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 02:07:44.836618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.842828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.849659 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.850586 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.864027 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.864259 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.928954 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.930109 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:44.936200 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:44.936618 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:44.938345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:44.955915 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:44.981330 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 02:07:45.042840 11144 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:45.042959 11144 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:45.043453 11144 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:45.043964 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:07:45.044560 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:45.044641 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:45.044718 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:45.044838 11144 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:45.167090 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.170075 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.175935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:45.175940 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:45.178395 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.178946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.292090 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.296078 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.302438 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:45.309293 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.319226 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.320003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.321142 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.329005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:45.332231 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:45.332929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.356839 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.359567 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:45.362725 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:07:45.441124 11268 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:45.441262 11268 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:45.441738 11268 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:45.442550 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:45.442842 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:45.442914 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:45.443116 11268 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:45.652488 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.662106 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:45.672014 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.672732 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.673455 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.676863 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.679186 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:45.686165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:45.692580 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.693710 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:45.699923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.734351 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.812355 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.816457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.817808 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:45.820010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:45.835966 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:45.839588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:45.930562 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 02:07:46.021144 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 02:07:46.032624 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.033438 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.042287 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.043215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.044605 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.060689 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.146951 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.147382 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.150177 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.157372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.161048 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.172958 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.255059 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.257953 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.267154 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.277734 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.277850 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.279603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.360388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.362134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.363550 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.364459 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.366601 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.380455 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.574609 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.576954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.579674 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.579803 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.590484 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.590687 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:46.605939 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:07:46.694165 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.694231 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.706039 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.708597 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.709224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:46.718459 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.721228 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.723313 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.730245 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:46.744464 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:46.746273 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:46.771564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:46.791725 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:07:46.792308 11144 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:46.792423 11144 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:07:46.792877 11144 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:07:46.794170 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:07:46.794309 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:46.794359 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:46.794485 11144 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:07:46.994343 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:07:47.064149 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.075453 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.075729 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:47.078290 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.078521 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.081409 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.101320 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.105429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.109198 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.116206 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.118947 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:47.143500 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:47.144336 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 02:07:47.218145 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.222025 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.223361 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.236382 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.238778 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.242955 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:47.315070 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 02:07:47.440191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.442623 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.449596 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.455426 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.459326 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:47.460619 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.548528 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.548830 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.555862 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.556613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.558868 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:47.572235 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.653391 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.653847 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.655519 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.656350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.659956 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.661058 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:47.756345 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.760751 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.761003 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.763731 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.764815 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.778251 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:47.956247 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:07:47.964025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.965863 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.966723 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:47.971673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:47.985723 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:47.986812 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.078807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.087441 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.089054 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.090610 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.106017 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.117408 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.130604 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.132390 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.135309 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.147922 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.149214 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.163856 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.444499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.455185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.457159 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.460846 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.463526 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.467635 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:48.483784 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240503 02:07:48.519826 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.525857 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.526320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.527901 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.530825 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.533833 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:48.564404 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 02:07:48.585947 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:07:48.619138 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.619704 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.624496 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.627666 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.633795 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.647962 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.777802 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.781581 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.784210 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.792423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.793990 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.796022 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:48.930866 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.933784 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.935717 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:48.936266 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:48.954737 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:48.964329 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.043478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:49.045022 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.045992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.051954 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.061491 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.069326 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:49.088815 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:07:49.127847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.129889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.132844 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.133970 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.142227 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.145705 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:49.240756 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:07:49.360795 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.364552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.365511 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.372807 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.379781 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.385550 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:49.453013 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.471261 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.476889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.478463 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.498457 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.513588 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.518741 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:49.525725 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.528244 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.529573 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.547454 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.560457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:49.709019 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 02:07:49.785524 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:07:49.786113 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240503 02:07:49.841296 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.842697 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:49.845019 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.856061 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.863657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:49.877524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.915050 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.915243 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:49.917554 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.917749 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:49.921976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:49.936918 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.027372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.030795 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.031206 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.032258 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.033821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.037173 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.179946 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.188941 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.189707 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.193493 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.194483 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.199525 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:50.325242 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:07:50.341885 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.345032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.345579 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.350347 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.350697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.359798 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.466194 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.466681 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.469926 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.482595 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.488373 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.488688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.543346 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.545991 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.550127 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.550472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.551909 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.569049 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.780853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.781687 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.786376 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.790766 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.799662 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.809446 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.887569 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.891013 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.895321 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.897922 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.908610 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.912871 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:50.950305 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.955673 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.958284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:50.959195 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:50.960625 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:50.970515 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:51.092887 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 02:07:51.150370 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 02:07:51.210386 13681 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:07:51.210505 13681 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:07:51.210986 13681 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:07:51.215162 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:07:51.216481 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:07:51.216569 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:07:51.216781 13681 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:51.217634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.218276 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.219781 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.222311 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.227936 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.250432 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.307404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.308602 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.309232 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.312085 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.312731 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.316118 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.409653 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.420646 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.422169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.422485 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.423070 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.441140 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:51.507242 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:07:51.540165 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.540680 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.544100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.545605 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.553893 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.556457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.730348 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.730753 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.732769 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.740077 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.745795 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.757212 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:51.860862 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:07:51.865430 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.868579 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.870038 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.883651 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.886116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.887454 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:51.889595 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:51.910564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.912561 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.914028 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:51.917459 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:51.943673 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:52.062040 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240503 02:07:52.242491 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.245836 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.250429 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.252360 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.252856 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:52.259960 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:52.277526 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.280061 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.296298 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.306516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.307607 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.308382 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:52.316571 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240503 02:07:52.392855 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.398195 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.401851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.402323 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.404099 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.408545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:52.512032 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:07:52.608825 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.608836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:52.611464 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.615962 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.618467 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.619084 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.714968 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.715417 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.715557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.718037 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:52.721762 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:07:52.725515 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.729568 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.823086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:52.823259 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.827387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.829113 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.838440 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.857774 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:52.933457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.935943 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:52.938001 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:52.938699 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.945669 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:52.954516 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.150202 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.150570 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.153992 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.154469 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.154913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.168866 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:53.219542 14529 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:53.219674 14529 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:53.220105 14529 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:53.220762 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:53.221045 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:53.221105 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:53.221318 14529 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:53.275139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.277804 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.278189 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.279458 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.285676 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.290817 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.303547 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.309335 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.311116 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.317029 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:53.320351 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.330408 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:53.382544 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 02:07:53.640331 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.648491 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.653461 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.654441 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.665822 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:53.671908 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.680328 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.689759 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.692123 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:53.699179 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.714278 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.720455 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:53.782439 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:07:53.792157 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 02:07:53.798976 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.800805 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.802114 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:53.813135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:53.821636 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:53.828716 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.019297 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:54.023351 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.027510 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.028143 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.042433 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.055887 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:54.088863 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:07:54.136019 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.136479 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.137115 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.142508 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.154774 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.162247 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:54.170202 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:07:54.243665 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.245216 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.248006 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:54.257117 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.257283 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.260660 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.350145 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.354635 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:54.356127 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.359094 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.367204 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.376752 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.565378 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.565896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:54.569300 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.577726 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.578213 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.588129 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:54.600140 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 02:07:54.687763 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.688548 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:54.691256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.697108 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.706750 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.707602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.721827 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:54.723234 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.729120 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:54.730193 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.734735 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:54.736256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:54.893131 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 02:07:54.915732 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:07:54.916414 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:07:55.056517 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.057157 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.062913 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.067260 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.071553 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.082229 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.085122 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.085353 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.103497 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.108520 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.108945 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.133955 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.205034 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.205755 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.206173 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.215291 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.215552 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.222646 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.422780 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.424983 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.425398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.432737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.435389 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.450583 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.524875 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.525048 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.526672 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.528462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.529994 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.555994 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.632463 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.634080 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.635636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.636046 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.645989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.657783 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:55.724452 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 02:07:55.743468 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.746047 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.747833 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.751914 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.756737 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.770216 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.940511 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:55.947562 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.950527 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:55.951084 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:55.952399 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:55.952777 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.069255 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.074400 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.078425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.080509 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.087283 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.114046 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.119174 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.121701 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.124126 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.140466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.142532 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.147352 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:56.407116 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 02:07:56.461740 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.467963 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.473256 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.478993 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.482065 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.488430 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.498845 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.502241 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.509022 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.517858 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.526320 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:56.540614 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:07:56.540853 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.607071 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.608246 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.608654 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.616003 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.617542 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.619007 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:56.796315 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:07:56.830422 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.830965 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:56.838027 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.838315 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.843202 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.847347 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.932869 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.934119 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.938529 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:56.944691 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:56.946601 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:56.950469 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.039408 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.044276 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.045354 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.045428 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.048319 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:57.066443 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:57.141599 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 02:07:57.151401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.152519 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.154963 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.156826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.157764 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.166455 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:57.354188 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 02:07:57.354744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.357507 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.358150 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.358472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.368537 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.379496 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:57.486233 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.488375 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.496469 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.506072 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.510181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.520530 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:57.528419 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:07:57.529314 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.530180 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.534112 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.545742 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.563781 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.564298 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:07:57.708989 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:07:57.855700 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.866143 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.872162 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.880710 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:57.882964 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.883447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.894210 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:57.911703 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:57.911705 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:57.915913 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.921875 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:57.922016 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.019224 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.023692 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.024310 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.026731 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.026821 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.032410 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.218286 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.221263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.222954 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.224432 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.233043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.234241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.334714 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.335539 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.340575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.341630 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:58.343591 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 02:07:58.344405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.359534 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.438647 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.442968 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.446967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.447394 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.466562 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.469269 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.545003 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.545565 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.546136 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.546319 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.560298 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.570349 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.598469 16341 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:07:58.598591 16341 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:07:58.598974 16341 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:07:58.599607 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:07:58.599931 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:07:58.599992 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:07:58.600107 16341 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:07:58.760612 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.764719 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.765307 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.770085 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.771379 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.777932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:58.853690 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 02:07:58.882782 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.890189 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.890686 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.896519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.897974 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.901620 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.904031 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:58.915004 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:58.920514 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:58.920957 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:58.930474 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:58.944521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:58.970063 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:59.141862 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:07:59.249694 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.252451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.261494 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.272557 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.275467 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.282320 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.322616 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.326500 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.331344 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.331743 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.346669 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.346979 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.424779 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.426942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.427531 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.429493 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.448139 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.452492 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:07:59.539180 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:07:59.544894 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 02:07:59.585489 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.586681 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.587010 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.598124 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.606698 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.607671 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.742633 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.745311 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:59.747824 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 02:07:59.748291 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.750089 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.750629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.765226 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.855301 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.864682 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:07:59.865231 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.865927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.872893 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:07:59.877820 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.941797 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.943022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:07:59.954651 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:07:59.958396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:07:59.964030 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:07:59.964536 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:07:59.964885 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:07:59.970428 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.182853 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.184428 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.186121 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.186926 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.190240 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.197034 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.309357 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.309690 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.309706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.320290 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.322548 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.326850 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.327214 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.337641 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.338483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.345456 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.379747 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.381219 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.677772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.684036 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.691545 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.692930 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.694190 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.714344 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.720865 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.725700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.731115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.733124 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.739007 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:00.741328 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.838514 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.841033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.845577 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:00.851121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:00.853022 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:00.868806 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:00.966480 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 02:08:01.045778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.045789 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.046682 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.048377 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.049414 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.074180 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.156700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.158398 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.160642 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.168951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.173497 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.177078 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.267593 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.269526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.278903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.281188 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.281744 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.284539 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.370512 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.373006 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.375960 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.377594 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.381868 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.397255 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:01.530310 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 02:08:01.584546 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.587208 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.589301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.589728 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.589849 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.589989 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.702598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.703111 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.706324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.716135 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.716457 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.731673 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.738353 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:01.747704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:01.749534 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.750170 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:01.758927 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:01.764698 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:01.833178 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:08:02.004982 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 02:08:02.080263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.086917 14781 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.089573 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.092554 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:02.097993 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.100329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.101142 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.101931 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.105086 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:02.121289 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:08:02.139783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.141007 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.142730 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.231508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.235155 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.242377 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.242923 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.243134 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:02.258632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:02.418406 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 02:08:02.442607 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.448418 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.459599 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.462260 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.466099 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:02.469225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.553157 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.553339 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.557408 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.563931 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.564267 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.567221 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:02.601424 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:08:02.657265 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.661187 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.661815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.661864 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.669749 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:02.676584 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.766594 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.767939 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.770000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.770874 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.782245 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.783938 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:02.965673 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:08:02.970415 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.972564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:02.972779 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:02.975224 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:02.981201 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:02.983409 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.101341 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.101749 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.102453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.102738 18194 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:03.102855 18194 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:03.103322 18194 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:08:03.107446 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:03.109050 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:03.112681 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:03.112757 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:03.112890 18194 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:03.113306 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:03.122399 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.128412 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.137351 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.140648 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.140817 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:03.151899 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.155045 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.476716 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.481559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.488194 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.492796 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.502918 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.506219 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.523279 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:03.523595 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:03.527920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.531282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.539952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.553789 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.630383 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.632298 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.637877 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.643999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.644377 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:03.656983 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 02:08:03.661260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.852703 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:03.855556 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.855859 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.856467 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.867942 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.882488 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.960748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.962661 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:03.963888 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.967743 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:03.970955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:03.981165 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:04.043998 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 02:08:04.069823 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.069933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:04.071518 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:08:04.072175 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.073961 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.074703 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.086325 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.177716 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.178009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.180560 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.181917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.183199 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.191557 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.257659 13681 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:04.257776 13681 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:04.258204 13681 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:08:04.259189 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:04.259683 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:04.259778 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:04.259918 13681 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:04.381639 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.382918 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.384290 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.388234 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.389434 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.397511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:04.475283 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 02:08:04.493779 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:08:04.510571 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.513976 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.516238 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.517041 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.525585 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.525857 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.531905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.539479 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.540706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.574266 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.575223 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.576625 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:04.653352 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:08:04.710158 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 02:08:04.889766 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:08:04.891281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.891847 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.901367 11144 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:04.901469 11144 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:08:04.901911 11144 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:08:04.906342 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.906601 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:04.908120 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.911929 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:04.913718 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.916303 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:04.924507 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240503 02:08:04.926086 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:04.939705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:04.943791 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:04.949632 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:04.949743 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:04.949795 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:04.950011 11144 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:04.966828 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.040184 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.044032 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.045642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.046768 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.048903 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.068732 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.259047 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.259343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.263577 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.269693 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.280898 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.295943 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:05.328032 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:08:05.369009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.369999 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.372192 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.380820 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.388669 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.390902 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.478703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.480226 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.480619 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.486675 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.488548 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.489920 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.589370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.590898 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.591231 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.594700 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.604507 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.606035 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.786690 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.786970 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.792433 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.801901 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.803823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.804260 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.919840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.921406 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.923132 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.928944 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.931346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.948311 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:05.950273 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.950311 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:05.959661 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:05.959807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.960806 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:05.961927 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:06.170467 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 02:08:06.288703 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.298578 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.298964 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.299404 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.299983 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.312111 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:06.326797 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.335916 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.336928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.368161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.373409 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.373512 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:06.379513 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 02:08:06.451215 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.452153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.459043 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.464967 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:06.472932 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.481987 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.666757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.669971 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.670017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.673033 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.673516 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.689060 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:06.771066 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.777011 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.777324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.778015 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.785079 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.786306 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:06.819399 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 02:08:06.882524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.886459 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.887244 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.895691 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.898370 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:06.911296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.983024 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.984908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:06.985499 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:06.988791 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:06.996618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.002022 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:07.080664 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 02:08:07.199158 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.200956 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.201274 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.204885 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:07.217499 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:07.221112 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:07.240401 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:08:07.308296 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.313156 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.319579 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:07.323523 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:08:07.324991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.329022 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.353071 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.355479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.360204 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:07.361517 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.370637 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.372812 14651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:07.373016 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:07.397706 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:08:07.697597 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.700700 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.701805 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.705195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.709388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.714290 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:07.714809 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:07.737475 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:07.745818 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.752758 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.756898 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:07.766150 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.848812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.850013 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.852274 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:07.864449 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:07.872299 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:07.878261 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.051445 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.055855 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.057417 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.057992 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.066794 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.068903 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.171069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.174654 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.175380 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.178695 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.179404 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.192543 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.279383 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.284453 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.287158 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.291234 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.297116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:08.298049 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:08:08.306937 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.383438 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.385644 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.391203 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.392558 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.396497 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.404353 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.598488 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.601686 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.603291 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.604271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.608147 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.637409 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.728689 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.730870 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.735577 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.743499 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:08.744266 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.744900 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:08.757835 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.764281 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:08.781108 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:08.783890 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 02:08:08.794698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:08.801963 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:08.816596 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 02:08:08.817463 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.077041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.079141 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.079182 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.082484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:09.082775 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.083425 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:09.147334 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:08:09.154353 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.159431 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.161695 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:09.170532 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.171077 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:09.175266 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 02:08:09.175945 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.261267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.263129 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.264294 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.267642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.272742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:09.284457 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:09.393368 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.394315 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.400012 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.404953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:09.411451 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:09.419199 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.576925 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.579602 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.579839 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.584879 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:09.585135 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.600461 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:09.683146 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.684433 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.687350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:09.696696 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:08:09.698910 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:09.701355 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.715886 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:09.760479 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:08:09.760573 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.761807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:09.763495 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:09.766762 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:09.776257 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:09.776515 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 02:08:09.780596 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:09.911375 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:08:10.009351 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.010566 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.011309 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.014596 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.019899 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.026768 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.123327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.131567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.131669 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.132401 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.140849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.164556 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.170639 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.175880 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.184420 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.187183 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.202958 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.215548 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.502658 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.506978 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.516466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:10.516698 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:10.519963 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.533507 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.535553 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.545168 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.557437 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.564918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.565564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.571193 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.572600 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.663852 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.664921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.669988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.672267 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.678789 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.691300 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.874189 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.875413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.876478 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.878422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.886615 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.893890 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:10.978065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.980645 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.981721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:10.983675 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:10.989717 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:10.996491 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.090425 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.095038 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.100533 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.103338 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.105607 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.113497 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.196583 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.202500 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.202978 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.204388 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.207926 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.208909 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.405947 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.411933 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.412859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.415402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.419821 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.426395 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:11.470626 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 02:08:11.527172 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.530678 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.544131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.549578 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.550297 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.552469 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:11.559410 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 02:08:11.568007 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.571899 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.588943 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.589977 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.591428 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.600097 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:11.911942 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 02:08:11.917762 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.920897 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.923084 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.928160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.931927 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.931927 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:11.932934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:11.945416 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.976502 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.977857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:11.980293 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:11.990628 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.096987 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.096987 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.103636 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.106609 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.113473 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:12.118682 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:08:12.120198 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:12.258073 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:08:12.287312 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:08:12.298992 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.300451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.302134 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.302794 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.311969 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.315984 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:12.335403 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 02:08:12.410895 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.411872 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.414160 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.415848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.418751 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033334
I20240503 02:08:12.422266 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.000s sys 0.002s Metrics: {"bytes_written":1983,"delete_count":0,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":78,"threads_started":1,"update_count":28,"wal-append.queue_time_us":150}
I20240503 02:08:12.429175 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.136223
I20240503 02:08:12.439000 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.439400 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.463091 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.008s sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14675,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":174,"lbm_read_time_us":299,"lbm_reads_lt_1ms":17,"lbm_write_time_us":261,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":121,"threads_started":2,"update_count":28}
I20240503 02:08:12.521724 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.523559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.525705 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.528548 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.539099 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.545348 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.619781 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.622447 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.623459 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.625451 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.629242 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.645285 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.839517 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.840152 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.842370 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.843909 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.846545 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.866231 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.966444 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.970786 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.971854 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.973346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:12.978523 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.980319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:12.981933 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:12.990514 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:12.991317 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.017803 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.022207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.035995 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.316346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.317564 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.318308 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.319236 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.325322 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.337720 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:13.381675 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.384997 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.385282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.386004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.386121 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:13.402457 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.472609 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.477165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.485471 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.485778 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.489101 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.504457 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:13.572517 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:08:13.698524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.699263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.702316 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.707037 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.708667 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.717362 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:13.769006 20243 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:13.769143 20243 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:13.769585 20243 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:08:13.770925 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:13.771025 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:13.771111 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:13.771322 20243 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:13.793685 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.798116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.800244 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.815112 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:13.815541 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:13.815793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.906010 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.909420 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:13.912704 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:13.912932 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:13.915858 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:13.918200 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 02:08:13.929591 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:14.007889 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 02:08:14.014034 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.018594 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.019979 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.020229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:14.036254 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:14.050017 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.223937 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.226215 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.226647 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.227458 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:14.240989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.246699 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:14.250861 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:14.339964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.348994 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.355469 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.365059 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:14.375846 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:14.377517 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.378537 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.382757 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.384232 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.387008 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:14.396726 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.405234 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:14.494843 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:08:14.523653 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 02:08:14.597385 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:08:14.694201 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.695564 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.700842 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.703161 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:14.704457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.718297 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:14.759542 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 02:08:14.766394 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.769630 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.778936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.780800 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.791464 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:14.792711 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:14.854805 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.856735 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.860371 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:14.868516 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:14.878690 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:14.879709 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:15.037279 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:08:15.098533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.098791 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.099457 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.112941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.113494 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.120255 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.201646 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.204912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.205027 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.206180 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.217013 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.220592 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.311877 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.312299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.312536 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.313308 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.316566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.324486 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.416028 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.419706 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.420557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.420774 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.423624 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.433051 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:15.527062 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:08:15.622124 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.626799 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.627945 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.628405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.629348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.634663 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.732599 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.737624 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.746488 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.752939 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.766582 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.770133 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.774538 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.777078 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:15.781692 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:15.801672 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:15.803556 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:15.817045 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.117812 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.123371 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.126645 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.138224 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.146713 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.166352 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.186636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.193279 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.196854 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.200985 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.204851 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.205461 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.294010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.297060 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.299387 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.300870 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.305341 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.311918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:16.339287 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 02:08:16.390561 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 02:08:16.509490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.509490 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.512109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.513769 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.516253 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.519979 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.612293 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.614612 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.616464 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.625294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.639283 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.639923 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.716495 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.720954 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.723162 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.726851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.732662 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:16.742542 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.826308 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:16.831897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.834144 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.835161 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:16.839615 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:16.846055 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:16.937661 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:08:17.003116 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 02:08:17.035887 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.036711 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.038626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.042771 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.056175 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.066816 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:17.155424 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:08:17.161110 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.169999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.171042 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.171568 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.177671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.192787 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.198551 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.199235 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.209740 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.226895 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.231122 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.231786 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:17.365361 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 02:08:17.517174 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.517855 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.519848 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.526365 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.540948 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.543793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.584023 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.585258 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.585836 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.592473 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.598480 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.601649 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:17.621662 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:08:17.661526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.666816 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.667425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.669068 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.669636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:17.669879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.909534 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.909745 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:17.915280 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:17.918704 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.919396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:17.920766 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.015308 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.017912 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.018978 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.021149 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.023666 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.049142 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:18.134406 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.137723 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.140913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.141446 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.147886 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.157230 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:18.239816 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.239902 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.241441 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.242996 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.246975 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.271243 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:18.333648 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:08:18.334398 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:18.453593 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.455091 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.462407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.463336 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.486439 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:18.486639 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.567680 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.582257 22031 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:18.582396 22031 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:18.582803 22031 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:08:18.583002 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:18.583591 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:18.585180 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:08:18.586462 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:18.586941 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:18.587023 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:18.587288 22031 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:18.593123 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.607141 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.610580 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.614985 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.625195 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.626703 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.633241 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.635578 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.636958 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:18.780179 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 02:08:18.874127 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 02:08:18.959074 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.964337 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:18.964452 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.966111 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.967711 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.973778 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:18.976940 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:18.988263 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:18.998364 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.013882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.023109 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:19.023885 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.113571 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.116758 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.119414 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.121666 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.141332 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.145023 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:19.263590 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:19.318806 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.322142 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.325399 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.325654 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.327463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.348891 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:19.351226 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:08:19.431977 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:08:19.433411 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.439213 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:19.440858 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.441115 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.443305 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.443967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:19.545126 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:19.549108 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.549582 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.550266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.553135 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.553567 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.555085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:19.641587 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.643594 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.646576 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:19.651996 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.664450 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:19.664845 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240503 02:08:19.681167 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:19.801177 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240503 02:08:19.871068 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.876128 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.876412 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.886178 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:19.892338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.892607 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:19.992137 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:19.993206 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:19.997455 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.008163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.010046 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.018990 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.051119 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.056013 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.061534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.064702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.080328 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.089279 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:20.316963 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:08:20.343883 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.345636 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.348048 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.348444 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.366205 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.367560 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.426057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.427925 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.437813 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.439365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.441195 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.449103 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.543510 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.546348 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.548202 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.549109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.552954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.559916 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.668123 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.671115 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.671501 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:20.674044 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.680546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:20.680745 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:08:20.686508 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.856156 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.856616 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.859483 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.861694 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:20.865744 22033 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:20.865862 22033 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:08:20.866350 22033 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:08:20.869580 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.877583 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:20.884492 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:08:20.892282 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:20.893019 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:20.893110 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:20.893246 22033 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:20.967896 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.968448 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:20.970459 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.988840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:20.996129 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.000533 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.039664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.040290 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.041988 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.048681 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.048940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.049012 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.302179 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.305449 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.311933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.311969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.313367 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.327522 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.396271 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.396751 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.398108 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.399938 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.406102 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.413467 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:21.436498 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 02:08:21.457283 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 02:08:21.464567 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.466087 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.481506 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.482137 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.484694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.502835 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.549468 23216 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:21.549594 23216 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:21.550057 23216 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:08:21.550818 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:21.550974 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:21.551049 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:21.551219 23216 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:21.733434 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.733892 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.735179 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.739446 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.740187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.740453 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.820506 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.825982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.828943 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.830848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.836426 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.842171 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.932368 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.932732 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:21.936036 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:21.939894 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:21.945255 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:21.955929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:21.971169 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:08:22.048686 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 02:08:22.057952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.060353 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.064452 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.064493 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.065075 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.076283 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:22.178961 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:08:22.250095 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:22.251247 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 02:08:22.255959 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.256677 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.258558 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.258987 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.267650 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.373834 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.374982 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.383225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.383942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.390201 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.398864 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.414681 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.416213 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.428299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.428889 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.434952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.466943 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:22.540057 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:08:22.737334 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.742789 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.744992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.747668 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.748515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.755394 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.811539 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.811837 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.824487 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.825803 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.826392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.851025 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.901793 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.902458 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:22.903604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:22.907655 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:22.909461 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:22.915905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.145239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.145998 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.146555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.153172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.154958 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.157200 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.242177 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.242172 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.245842 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.247524 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.255923 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.256181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.348062 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.348659 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.354482 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.360149 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.360697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.367921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.461882 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.463279 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.463935 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.464401 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.468041 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.474891 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.656391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.659426 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.662004 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.663667 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.667955 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.673712 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.768913 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.769739 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.788197 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.793056 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.794440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.799998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:23.801777 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.803028 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.822311 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:23.829558 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:23.836148 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:23.836664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:23.856628 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 02:08:23.858847 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:08:24.033459 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 02:08:24.164727 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.169008 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.182737 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:24.183676 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.184913 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.186374 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.187835 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.188004 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:24.188664 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.197968 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:24.205226 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.224316 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:24.305744 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:08:24.319331 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.326503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.327534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.327790 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.328037 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:24.350914 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:24.512609 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:08:24.520723 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.521011 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.521986 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.523670 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:24.537163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.538702 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:24.568873 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 02:08:24.632318 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.633059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.634435 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.635499 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:24.637863 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 02:08:24.642791 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:24.651103 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.732676 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.735220 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.741770 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:24.742462 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:24.752836 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.755961 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:24.763432 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:08:24.834797 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.837821 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.839861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:24.843525 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:24.846915 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:24.864303 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.055404 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.057389 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.057803 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.063910 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.064281 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.073190 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.160485 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:25.162881 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:08:25.165030 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.165760 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.166918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.175359 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.195246 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.223520 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.228552 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.229766 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.232769 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.234858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.236543 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.490025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.490810 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.492364 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.492965 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.504843 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.520273 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.578896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.579967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.581590 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.583213 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.584600 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.601673 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.685842 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.686033 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.688833 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.690465 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.691962 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.692314 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:25.705364 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:08:25.802358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:25.803627 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.803905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:25.804777 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:25.807906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:25.837291 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.002864 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.007941 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.010253 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.011051 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.011829 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.022946 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.141867 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.145031 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.154196 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.154897 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.155403 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.155562 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.156827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.157048 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.157950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.178869 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.179522 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.202693 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:26.296932 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 02:08:26.504132 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.510951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.511721 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.512460 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.514536 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.537320 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.560668 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.563520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.564235 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.573338 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.575238 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.604085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.651078 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.654469 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.656239 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.662182 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.663772 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:26.669945 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 02:08:26.671133 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.890316 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.892786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.896096 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:26.896276 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.909420 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:26.915308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.988970 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.990509 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:26.992551 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:26.994233 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.000568 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.013830 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:27.058333 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 02:08:27.098278 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.099033 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.103194 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.111491 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.127146 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.136215 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:27.185624 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 02:08:27.209609 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.209988 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.212888 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.215715 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.219310 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.225109 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:27.273840 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:08:27.379127 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:08:27.420392 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.422710 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.423205 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.425037 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.436450 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.437434 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.551667 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.552145 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.554713 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.556456 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.557565 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.564227 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.579725 23216 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:27.579856 23216 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:27.580278 23216 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:08:27.581905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:27.582527 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:27.584599 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:08:27.584669 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:27.584892 23216 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:27.589867 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.594954 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.607894 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.613095 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.621136 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.912999 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:27.913379 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:08:27.921710 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.925211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.926116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.928627 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.946273 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.950621 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:27.954437 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:27.957949 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.962332 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:27.967443 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:27.991863 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.069113 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.069308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.070613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.071903 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.073117 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.074753 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.280644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.285406 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.287890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.290987 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.291287 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.296284 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.382290 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.385776 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.387434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.390089 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.400508 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.402267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.498298 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.501693 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.502761 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.503183 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.509591 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.519490 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.599275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.601613 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.605394 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.606725 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.607101 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.636703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.827535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.828356 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.834164 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.836038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.839524 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.852041 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:28.932961 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 02:08:28.947624 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:28.948735 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:08:28.948879 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.959038 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.959508 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:28.970806 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 02:08:28.970932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.973429 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.973784 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.981253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:28.983683 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:28.990422 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:28.991171 12756 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:28.991909 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:29.299820 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 02:08:29.312301 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.314038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.321931 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.326215 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.332918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:29.340196 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.344025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.346101 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.347303 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:29.348702 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:29.351848 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:29.372253 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.392606 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.464311 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.465304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.471956 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.473632 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.475708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.484206 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:29.582581 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 02:08:29.582957 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:08:29.677398 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.680070 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.682866 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:29.682898 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.694535 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.705551 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.780936 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.783471 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.785615 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.790905 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.793823 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.802817 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:29.894781 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.897018 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.900177 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:29.904569 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:29.910553 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:29.925873 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.006388 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.008028 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.009004 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:30.010107 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:08:30.010808 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.022639 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.046422 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.210994 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.214188 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.216542 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.217700 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.220327 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.220809 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.321863 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.322976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.336411 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.336990 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.337994 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.339290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.388607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.392112 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.394042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.394709 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.397092 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.404764 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:30.599869 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:08:30.650836 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.652084 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.652365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.654623 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.655340 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.676765 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.737618 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:30.742213 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.743345 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.746613 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.754297 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.754315 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.811973 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.815613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.815785 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:30.829263 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:30.832137 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:30.841742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:30.857972 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:31.054728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.057443 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:31.060341 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.068188 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.070750 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.080850 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.154364 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.154670 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:31.158078 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.160806 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.180812 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.184790 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.264389 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.268849 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.275405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.276759 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.279083 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.279747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:31.293614 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 02:08:31.367157 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.371129 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.375094 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.377478 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:31.391599 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.392979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:31.416289 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 02:08:31.585526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.588517 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.591921 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.593050 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.594687 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:31.617264 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.705695 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.709059 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.715210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.716610 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.719794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:31.728051 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 02:08:31.731653 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.739459 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:31.751102 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.751801 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:31.762660 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:31.769037 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:31.771211 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:31.930065 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 02:08:32.083899 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.087318 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.089890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.092506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.093995 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.094797 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.095969 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.107692 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.131711 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.131739 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.132014 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.132421 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.196559 23216 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:32.196732 23216 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:32.197294 23216 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:08:32.198158 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:32.198750 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:32.198822 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:32.198966 23216 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:08:32.210755 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:08:32.232697 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.235472 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.243907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.244418 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.245505 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.270598 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:32.353433 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:08:32.430682 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.432581 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.435510 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.436750 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.437134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.462471 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.554248 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.558105 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.558624 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.564147 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.569872 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.577914 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.654413 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.656160 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.656808 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.657313 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.658102 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.673161 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.760668 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.760641 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.761968 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.772467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.777076 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:32.777226 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.960536 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.967116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:32.967604 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.968385 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:32.969348 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:32.969822 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.074954 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.086963 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.089519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.090015 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.090335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.099218 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.115420 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.126123 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.131649 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.134690 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.138105 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.159552 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:33.177939 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:08:33.463197 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.475036 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.477815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.483316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.487210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.487352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.495354 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.504060 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.505406 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.508878 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.518558 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.539005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.628486 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.632009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.634914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.636591 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.649232 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:33.653086 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:33.722271 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 02:08:33.843108 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.844707 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.844872 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.853171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.870343 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.872824 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:33.909039 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 02:08:33.961297 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:33.964618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.965631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.967396 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:33.976403 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:33.979013 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.064885 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.068228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.072189 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.073455 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.074445 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:34.090570 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.164966 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.165903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.166702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.168594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.176256 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:34.189126 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:34.297755 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:08:34.384130 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.384550 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.385742 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:34.387823 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240503 02:08:34.390805 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.395228 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:34.396961 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 02:08:34.407742 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.494211 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:34.494661 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 02:08:34.496390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.499990 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.501734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.502549 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.511518 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:34.568817 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.570407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.573892 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.574918 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:34.576803 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.579926 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.811529 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.813019 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.815153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.824757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.825191 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:34.828943 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:34.855054 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:08:34.861315 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:08:34.923266 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.925972 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.926038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:34.926915 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:34.930528 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:34.942834 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.024932 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.027086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.033823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.039840 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.039927 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.043937 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.140616 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.141140 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.142797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.151712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.152180 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.164337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.258365 26084 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:35.258483 26084 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:35.258927 26084 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:08:35.260016 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:35.260116 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:35.260206 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:35.260344 26084 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:35.337944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.338187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.342479 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.346323 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.347923 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.350659 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.455631 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.457048 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.458302 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.460831 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.500422 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.505002 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.505024 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.517611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.517680 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.520025 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.520599 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.536723 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:35.623539 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:08:35.832126 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.833940 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.849560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.865041 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.865782 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.867277 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.897847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.899869 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:35.907001 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:35.909516 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.928545 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:35.931924 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.998014 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:35.999579 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.000589 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.001189 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.025189 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.028797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.031916 26041 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:36.032044 26041 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:08:36.032519 26041 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:08:36.033371 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:08:36.034216 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:36.034294 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:36.034376 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:36.034507 26041 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:08:36.118577 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:08:36.223415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.228526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.228677 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.228794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.229511 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.239087 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:36.283317 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 02:08:36.344820 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.345409 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.354595 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.358222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.360539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.361258 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:36.390425 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 02:08:36.454530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.457783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.458173 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.476738 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.479410 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.480072 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.558403 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.560025 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.560321 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.565953 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.566144 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.587353 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.771477 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.776021 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.777920 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:36.778087 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 02:08:36.780064 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.790309 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.791574 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.892129 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.892630 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.895201 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.896896 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.904338 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.907949 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:36.936009 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 02:08:36.956624 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.958045 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:36.960227 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:36.960527 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:36.963918 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:36.968467 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.221884 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.225843 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.226365 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.227299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.238009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.269197 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.310770 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.314116 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.320151 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.321574 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.325954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.327323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.394253 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.394701 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.394961 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.399142 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.408222 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.410027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:37.477576 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:08:37.490406 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:08:37.638693 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.640038 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.642997 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.647923 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.651227 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.651979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.740402 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.742456 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.748308 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.749451 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.758514 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.759902 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.854182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.854359 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.855237 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.865599 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.866014 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.888182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:37.965996 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:37.967525 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.969535 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:37.970714 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:37.977665 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:08:37.985273 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:37.994310 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.175194 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.177512 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.177712 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.180892 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.185106 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.187031 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.300268 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.300731 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.302384 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.303112 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.304445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.309085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.345613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.352290 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.354523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.355573 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.358906 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.365217 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.637790 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.643055 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.654985 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.655596 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.656633 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.680423 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.721436 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.722622 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.731539 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.732441 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.737110 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.737363 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.806553 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:38.809445 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.813535 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:38.821611 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:38.822660 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:38.836788 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:38.849850 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 02:08:38.870373 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 02:08:39.037928 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.044224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.048626 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.050067 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.053547 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.054939 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.091367 28915 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:39.091490 28915 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:39.091876 28915 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:08:39.094719 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:39.095075 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:39.095155 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:39.095288 28915 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:39.152952 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.154029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.156919 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.158828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.175944 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.179401 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.263624 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.263919 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.265496 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.266471 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.272500 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.277514 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.366828 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.368850 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.373466 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.374801 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.382386 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.382897 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:39.413731 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 02:08:39.495163 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:08:39.495896 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 02:08:39.499673 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240503 02:08:39.577955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.581924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.586957 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.594048 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.596071 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.603324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.695225 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.705545 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.707381 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.715967 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.717139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.717579 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.727731 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.738021 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:39.754240 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:39.755236 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:39.756269 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:39.764870 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:39.963439 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:08:40.076828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.077538 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.080845 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.089679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.090164 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.095965 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.098423 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.099987 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.104251 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.116310 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.137203 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.142020 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:40.218292 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:08:40.226275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.227175 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.229122 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.229395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.234587 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.248793 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:40.375639 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:08:40.440728 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.442723 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.443797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.446827 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.466482 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.468452 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.555508 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.555599 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.558640 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.560030 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.560765 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.570917 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.663664 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.664019 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.664551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.668381 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.673429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.688712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.763633 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.770638 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.773535 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.776827 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.780803 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.791077 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:40.974035 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.974613 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.980116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:40.982504 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:40.989802 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:40.994037 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.096372 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.099586 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.100466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.103996 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.126960 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.129333 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.130790 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.140363 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:41.150568 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240503 02:08:41.150671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.152228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.154557 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.160658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:41.286762 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:08:41.482949 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.485299 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.485460 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.496865 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.498659 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.532061 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.535686 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.536281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.542042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.543587 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.553745 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:41.558518 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 02:08:41.577888 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.655072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.656149 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.657006 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.665771 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.666146 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.684182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:41.702589 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240503 02:08:41.829023 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.830242 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.837193 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:41.841574 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.849627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.861775 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.970342 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.971127 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:41.976485 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.977568 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:41.981150 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:41.981467 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.074929 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.079870 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.081400 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.081904 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.088936 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:42.094771 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:42.104431 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240503 02:08:42.180233 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.181548 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.184790 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.191648 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.200255 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:42.208712 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.390348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.394577 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.395030 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.398674 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:42.400236 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.412026 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.508327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.510095 29163 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:42.510219 29163 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:42.510629 29163 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:08:42.511373 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:42.512305 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:08:42.512660 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:42.512741 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:42.512864 29163 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:42.521245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.527671 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:42.533509 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.537103 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.559525 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:42.561074 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:08:42.563302 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.572562 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.574363 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.577518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:42.578177 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:08:42.592849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:42.850515 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.850869 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.854547 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.867877 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.870776 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.884887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:42.913807 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:08:42.934222 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:42.939247 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.944721 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.946048 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:42.952924 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:42.953606 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.016866 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.017372 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.019168 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.026816 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.028656 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.036111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.251129 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.252499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.253260 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.254840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.267397 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.267521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.360191 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.361603 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.362565 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.362811 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.362933 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.378890 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:43.426291 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 02:08:43.464720 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.468545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.468780 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.471325 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.474337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.474959 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.572211 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.572963 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.574664 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.578442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.585711 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.603367 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.784499 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.785163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.788184 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.789920 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.790827 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.808032 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.904903 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.906193 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.917367 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.920452 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.924219 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.926484 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:43.933286 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.935686 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.940788 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:43.951573 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:43.958650 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:43.984078 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:44.096238 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 02:08:44.216095 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 02:08:44.291343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.292165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.293632 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:44.303203 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.306947 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.310143 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.310830 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.331302 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.331537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.338126 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.345041 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.345155 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:44.440747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.444306 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.454888 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.458894 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.460644 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:44.468210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:44.511021 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:08:44.511952 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 02:08:44.606700 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 02:08:44.643620 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.646440 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.647260 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:44.647267 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.648021 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.661290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.763700 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.765674 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.766876 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.772987 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.773106 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:44.773355 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.876595 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.879957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.880373 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.889678 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.892398 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:44.909227 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:44.984198 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:08:44.984500 28501 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:44.984616 28501 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:44.984704 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.985100 28501 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:08:44.986613 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:44.987125 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:44.987205 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:44.987351 28501 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:44.987644 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:44.988325 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:44.992074 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.003005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.003005 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:45.063639 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:08:45.201133 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.204156 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.210608 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.211927 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.212587 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.221060 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.323590 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.334177 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.335978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.336383 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.348950 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.350014 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.350030 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.365435 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.369578 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.371796 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.379629 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.387372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:45.418560 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:08:45.704010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.704591 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.704849 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.705376 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.712872 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.718894 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.721738 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.743235 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.757059 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.759881 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.764524 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.764783 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.855401 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.856200 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:45.861642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:45.871800 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:45.871953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:45.879638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:45.961865 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240503 02:08:46.066646 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.067366 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.071197 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.073603 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.075718 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.092111 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.172747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.175742 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.180513 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.188071 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.188345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.189219 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.280330 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.281913 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.282593 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.297488 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.299000 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.299041 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.405705 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.409016 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.410459 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.412494 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.414098 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.418579 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:46.515959 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:08:46.516693 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240503 02:08:46.599500 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.603824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.606645 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.609705 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.610100 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.621096 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.712899 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.715050 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.722764 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.723745 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.726150 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.737336 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.772327 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:46.772976 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.778326 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:46.782460 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:46.783172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:46.789942 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:46.793134 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 02:08:47.045095 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.048348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.050516 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.057771 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.062374 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:47.069124 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240503 02:08:47.086620 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.136060 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.138373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.141539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.145826 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.151954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.173244 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.218217 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.219659 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.219996 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.225670 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.235006 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.240490 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.436718 30792 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:47.436851 30792 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:47.437274 30792 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:08:47.438375 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:47.439076 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:47.439141 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:47.439630 31401 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:08:47.450549 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:08:47.466048 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.466790 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.469655 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.474308 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.477437 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.480870 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.572233 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.575471 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.577791 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.580091 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.588943 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.601588 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.679275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.679275 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.680080 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.680645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.681531 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:47.695777 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.780652 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.781474 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:47.784054 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.784173 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:47.796990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.797730 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:47.918680 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:08:47.987406 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.991156 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.991722 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:47.999917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.000452 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.011538 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:48.080369 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:08:48.110986 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.116406 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.117496 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.122284 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.125747 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.126189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.126917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.142952 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.152379 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.161340 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.170545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.183228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.492053 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.500713 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.501211 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.504736 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.510674 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.513010 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.513530 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.514302 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.515965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.517684 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.528818 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.530040 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:48.546197 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 02:08:48.645804 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.650068 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.650735 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.653440 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.654554 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.662793 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.848137 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.848558 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.849187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.854755 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:48.854804 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.865041 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:48.932858 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 02:08:48.960024 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.963133 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.964704 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:48.966701 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:48.973701 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:48.974742 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.064162 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.066316 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.072751 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.075328 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.075904 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:49.081230 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240503 02:08:49.093703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.170743 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.171020 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.173727 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.177531 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.178752 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.188000 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.384227 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.385435 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.387303 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.388619 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.397146 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.406311 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.502662 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.509150 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.509557 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.517534 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.525195 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.533708 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.534093 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.541414 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.550068 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.558907 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.562068 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.584903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:49.608976 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:08:49.805315 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 02:08:49.868727 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.873333 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.873396 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.882774 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.892510 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.913635 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.939940 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.942451 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:49.945297 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:49.953382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:49.958755 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:49.961292 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.055395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.055971 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.056650 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.059281 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.060451 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.069610 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:50.071431 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:08:50.072314 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:08:50.225456 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.226096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.227223 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.228617 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.229267 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.240717 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.364792 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.366586 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.366914 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.369200 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.378885 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.384094 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:50.438040 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:08:50.468688 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.471921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.472530 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.476380 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.478729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.491807 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.560520 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.564424 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.568969 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.569500 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.570298 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.577395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:50.731180 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240503 02:08:50.738391 30742 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:50.738523 30742 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:08:50.738973 30742 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:08:50.739656 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:08:50.740501 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:50.740590 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:50.740666 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:50.740787 30742 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:08:50.745399 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:08:50.799139 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.799297 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.800741 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.803495 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.818579 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.832636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.919157 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.928942 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.930725 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.931131 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.935858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.940659 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.942265 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.949929 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:50.961191 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:50.966153 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:50.981009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:50.990742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.278626 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.291353 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.296260 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.301981 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.313630 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:51.314311 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.335466 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.336817 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.339679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.341723 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:51.350772 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.355921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.443920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.446799 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.447250 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.457465 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.458984 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.459234 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:51.496709 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 02:08:51.644662 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:08:51.645578 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 02:08:51.649726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.649825 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.651258 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.659968 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.676645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.678542 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:51.761847 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.766728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.767537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.769485 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.776139 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.779088 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:51.873618 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.876442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.885625 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.888449 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:51.895957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.899158 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.986678 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.987107 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:51.987720 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:51.988138 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:51.990414 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.011799 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.116705 31760 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:08:52.116876 31760 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:08:52.117321 31760 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:08:52.118091 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:08:52.118436 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:08:52.118502 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:08:52.118676 31760 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:52.191107 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.194656 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.194968 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:52.199196 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240503 02:08:52.200029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.218657 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:52.219975 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.321830 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:52.321918 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.326323 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.327245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.331979 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.334856 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.336541 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.338568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.351364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.367522 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.370560 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.375051 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:52.438743 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:08:52.563341 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:08:52.686877 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.691696 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.699081 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.700021 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:52.707464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.721606 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.727645 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.734414 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.736438 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.742391 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.762522 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.764686 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:52.843283 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.843361 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.847204 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:52.848626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:52.849041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:52.850566 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.062935 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.063570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.066322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.073807 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.074072 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.084233 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:53.170665 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:08:53.171221 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.172441 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.173046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.173621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.177336 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.188966 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:53.208645 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 02:08:53.277859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.285528 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.293421 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:53.293632 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.304971 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.306828 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.389597 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.389627 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.391541 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:53.393519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.395275 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.412175 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.603034 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.603649 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.605286 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.605767 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.609493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.627166 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:53.725976 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.732481 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.734171 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.738186 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.746083 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.754119 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:53.755970 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:53.757982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.770009 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:53.778183 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:53.800586 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:53.801033 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.085973 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.089337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.093840 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.098238 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:54.102408 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.121227 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:54.128367 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 02:08:54.146438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.158259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.158710 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.159869 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:54.162072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:54.187235 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 02:08:54.188140 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.269395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.273284 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.274840 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:54.277405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.280074 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.287436 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.435005 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.437531 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.440739 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.442436 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.452978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.456135 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:54.589490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.590967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.592247 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.592589 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.593570 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.609580 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:54.691226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.695358 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.699548 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.701771 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:54.709053 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:08:54.711810 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.723276 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.786285 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.792408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:54.794260 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.794502 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:54.802184 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:54.803509 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:54.874531 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:08:54.938165 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 02:08:55.007239 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.011894 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.013442 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.016259 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.019912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.023113 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:55.045562 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:08:55.136126 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.142426 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.145165 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.157169 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.169374 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.170859 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.172617 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.173403 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.190726 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.197695 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.199076 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.210820 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:55.250911 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:08:55.518711 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.520946 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.530119 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.534168 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.534603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.539690 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.546461 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.550117 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.551169 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.555567 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.569416 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.586866 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.669298 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.670378 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.670945 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.671104 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.681017 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.684655 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:55.724802 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 02:08:55.884932 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.890872 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:55.891916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:55.899983 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:08:55.903439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:55.903790 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:55.904951 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.998816 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:55.999887 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.000762 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.002190 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.009774 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.010111 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.080231 1331 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:08:56.080363 1331 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:08:56.080816 1331 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:08:56.082482 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:08:56.082605 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:08:56.082695 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:08:56.083317 1543 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:08:56.110069 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.111677 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.112639 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.113250 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.114450 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.114464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.207253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.211292 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.211741 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.217499 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.226003 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.237187 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.445785 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.446166 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.458211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.460004 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.460597 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.461570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:56.476136 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 02:08:56.572737 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.573513 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.574239 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.580902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.581375 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.588712 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.591002 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.592643 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.616571 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.619863 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.621843 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.625635 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:08:56.674335 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:08:56.674965 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240503 02:08:56.914762 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:56.916038 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.926817 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.926997 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.930371 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.949033 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.966979 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.970719 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.976998 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:56.977862 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:56.982182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:56.999838 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.085182 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.086913 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.090236 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.090332 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.099905 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.102460 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:57.284229 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:08:57.291674 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.292706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.298142 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.300369 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.301067 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:57.305061 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.397418 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.400949 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.402573 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:57.409790 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.410953 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.413724 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.509358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.509888 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.510821 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.519111 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.522549 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.534767 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:57.535337 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:08:57.599140 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 02:08:57.609015 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.612803 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.613457 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.621762 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.626353 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:57.636209 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.820022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.825726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.829782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.831195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.851617 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:57.858683 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:57.961755 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:57.961844 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.966424 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.969173 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.974403 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.974704 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.981528 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:57.987480 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:57.988564 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.001171 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.019232 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.025800 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:58.074503 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240503 02:08:58.314844 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.315392 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.326870 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.329847 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.336405 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.336663 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.368108 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.372982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.377718 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.378564 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.378994 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.393733 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:58.424337 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:08:58.477186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.478032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.479462 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.494747 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.495587 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.506970 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.672972 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.676257 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.677593 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.679189 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.686424 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.696053 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.794438 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.795713 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.799088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.803181 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.815064 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.818035 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:58.906690 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.911228 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:58.914355 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:58.918120 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.924361 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:58.927976 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.007877 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.009065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.015168 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.016717 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.017930 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.020239 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:08:59.177592 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240503 02:08:59.219158 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.219899 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.221644 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.221995 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.222333 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.237574 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:08:59.338665 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240503 02:08:59.342985 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.344864 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.345232 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.350479 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.352941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.363243 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.369765 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.376880 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.381289 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.389365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.392793 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.415248 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:59.690328 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:08:59.695336 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.704087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.705602 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.706964 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.707616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.714619 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:08:59.727838 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:08:59.756207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.757117 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.758049 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.770927 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.783878 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.791849 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.865798 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.871480 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:08:59.874087 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:08:59.883523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:08:59.885310 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:08:59.885576 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.043118 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.046483 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.050153 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.050849 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.054970 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.063707 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:00.071532 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240503 02:09:00.186874 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.190587 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.193694 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.195552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.212051 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.215911 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:00.219305 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:09:00.296082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.296396 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.306751 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.309427 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.310528 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.310520 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.394174 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.397253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.398134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.408713 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.409641 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.419595 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:00.588454 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:09:00.608323 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.608835 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.614889 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.618588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.627745 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.636248 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.721292 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.724356 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.732605 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.734380 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.742774 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.748633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:00.758683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.761243 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:00.768713 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240503 02:09:00.776590 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:00.777901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:00.784890 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:00.794189 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.085932 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.091691 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.096000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.101405 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.113077 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.114710 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:01.123255 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:09:01.148784 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.156442 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:01.156523 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.162708 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.163623 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.181334 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.255615 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.261510 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.266064 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.267730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:01.268697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.269579 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.419134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.420919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.421474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.423382 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.425554 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:01.435508 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:01.450953 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240503 02:09:01.571121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.572160 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.574991 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.574985 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.599969 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:01.600487 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.681015 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.682874 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:01.685031 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.686470 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.701402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.712877 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:01.774183 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.774327 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.778461 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:01.780902 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:01.782680 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:01.800658 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:09:01.802340 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240503 02:09:01.816362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.004331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.005956 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.005990 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.008837 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.009017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.020610 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.115872 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.117621 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.120767 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.130407 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.131651 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.150763 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.171627 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.180902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.184134 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.185966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.190904 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.205693 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:02.322000 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:09:02.457818 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:02.458093 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240503 02:09:02.460498 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.466877 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.470225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.476789 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.479971 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.543946 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.545643 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.552747 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.552755 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.554301 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.564908 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.651556 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.652276 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.659381 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.666558 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.670209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.676530 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.787858 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.788666 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.789330 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.790074 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.793641 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.808647 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.962677 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.965631 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:02.967967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:02.974618 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:02.979813 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:02.996826 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:03.055824 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:09:03.083201 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.086772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.091243 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.095894 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.097538 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.109079 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:03.158025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.159000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.161537 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.162132 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.164124 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:03.173238 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.409888 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.419845 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.426322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.428699 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.429232 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.435845 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:03.444737 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240503 02:09:03.515475 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.519958 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.521772 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.529564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.534162 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:03.540467 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:09:03.544473 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.582650 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.589501 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:03.591498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.593720 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.601078 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.616303 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:03.754158 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240503 02:09:03.793993 3742 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:03.794124 3742 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:09:03.794580 3742 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:09:03.795084 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:09:03.795809 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:03.795884 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:03.795938 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:03.796044 3742 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:03.856654 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.857544 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.865909 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.866631 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.877100 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:03.894147 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:03.951628 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:03.953312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.954126 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.954521 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:03.964489 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:03.973031 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.057032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.057135 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.059741 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.060474 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.063076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.073040 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.180075 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.181159 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.185870 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.186007 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.186961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.201586 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.367524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.367563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.372538 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.375949 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.376992 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.383134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.482110 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.486207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.487181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.487891 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.498196 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.503046 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.553905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.554195 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.558555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.559373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.568342 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.571450 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:04.593631 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240503 02:09:04.804854 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.807330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.808050 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.808244 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.812009 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.819607 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:04.857547 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:09:04.909482 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:09:04.927405 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.933203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.942305 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.943603 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:04.944075 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.952535 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.964038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:04.972383 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:04.973852 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.984047 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.987933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:04.996300 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:05.007869 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240503 02:09:05.292148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.297641 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:05.299772 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.299890 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.313098 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.314011 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.360220 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.360558 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.360764 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:05.363519 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.384171 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.394490 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.473263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.474292 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.482182 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:05.483325 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.490255 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.492467 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:05.509656 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:09:05.629877 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.630586 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.635591 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.636310 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:05.645215 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:09:05.650633 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.658242 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.783326 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.784076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.785300 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:05.790742 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.792790 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.800254 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.894645 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.896272 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.897030 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:05.898517 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.901540 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.928800 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.975034 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.980793 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:05.980903 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:05.989434 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:05.994203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:05.994272 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240503 02:09:05.994546 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.146899 2577 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:06.147044 2577 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:06.147581 2577 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:09:06.148247 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:09:06.148517 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:06.148582 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:06.148758 2577 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:06.219911 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:06.220943 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:09:06.224423 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.226248 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.227589 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.228525 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.229665 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.327970 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.331225 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.340726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.344338 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.356886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.374644 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.376194 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.384724 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.399477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.401293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.406723 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.407411 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:06.480381 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240503 02:09:06.666057 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.671326 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.671420 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.674329 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.691866 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.692895 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.760530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.760895 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.761531 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.761564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:06.765094 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.772655 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.866724 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.870270 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.871919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.872068 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.884493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.894166 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:06.963289 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:06.976855 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.981356 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:06.983558 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:06.983851 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:06.996847 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.008535 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:07.037012 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240503 02:09:07.179610 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.182324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.184623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.185575 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.187901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:07.192870 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:09:07.210091 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.311921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.314381 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.319943 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.322276 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.327311 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:07.341985 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.353801 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.364750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.369669 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:07.370460 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.370929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.374246 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:07.397992 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240503 02:09:07.680151 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.686358 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.694098 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.697091 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.699250 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:07.707589 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.710666 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.725764 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.726665 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.732391 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.736560 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:07.738211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.838735 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.839720 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.845240 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:07.846849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:07.858824 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:07.860646 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.037232 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.039450 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.041795 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.042479 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.045435 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.060303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:08.145813 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:09:08.152141 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.152256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.153159 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.160146 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.163224 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.179294 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.257771 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.258420 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.258477 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.269687 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.279222 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.280889 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:08.361285 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240503 02:09:08.369017 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.372819 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.373013 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.381132 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.382228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.383425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.574985 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.578395 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.578461 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.583886 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.584463 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.592900 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.679260 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.681434 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.687108 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.689656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.696743 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:08.698668 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240503 02:09:08.711174 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:08.744818 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:08.746286 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.747994 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.752559 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:08.757525 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:08.772413 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:08.896508 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:09:09.039275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.046674 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.050621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.052349 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.062309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.064079 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.108763 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.114449 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.115111 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.119395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.120893 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.153039 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.199576 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.200059 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.201324 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.201396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.202852 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.226222 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:09.353809 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240503 02:09:09.437280 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.438907 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.440291 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.444175 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.462535 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.472872 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.538962 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.540122 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.542088 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.544246 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.556238 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.566262 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.644519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.645721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.647339 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.648281 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.660708 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.663599 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.684329 3812 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:09.684450 3812 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:09.684841 3812 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:09:09.685863 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:09.685947 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:09.686040 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:09.686161 3812 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:09.756449 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.756626 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.760587 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.765636 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.769208 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:09.773886 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:09:09.779944 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.964555 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:09.964684 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.965318 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:09.969823 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:09.981770 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:09.981930 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:10.060875 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:09:10.071938 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240503 02:09:10.080303 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.099570 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.100881 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.104190 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.104362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.122162 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:10.124327 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.131894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.148025 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:10.149832 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.154776 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.155369 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.475735 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.475843 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.484824 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:10.486336 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.486443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.491235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.493425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.511086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.511461 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.529126 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.550041 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.551093 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:10.620852 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:09:10.628531 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.631994 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.632812 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.633534 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:10.633793 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.635090 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:10.671541 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240503 02:09:10.821970 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.822732 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.827211 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:10.830710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.836170 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.857748 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.933660 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.938609 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:10.940218 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.942456 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:10.948467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:10.953339 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:10.969571 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:09:11.039458 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.047595 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.052637 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.053189 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.056043 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.064245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:11.098485 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:09:11.109023 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240503 02:09:11.146664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.147558 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.152005 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.156847 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.160660 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.170889 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.356082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.356565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.360662 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.363912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.364439 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.366639 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.471551 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.481246 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.489897 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.491988 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.493433 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.494206 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.494880 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.496814 4942 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:11.496901 4942 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:11.497277 4942 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:09:11.502712 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:11.504189 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:09:11.504256 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:11.504438 4942 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:11.516537 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.520026 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.523131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.534700 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.537552 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.843422 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.846419 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.850503 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.868942 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.870956 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:11.883396 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.894568 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:11.896850 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.897612 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.905386 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:11.914151 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:11.921712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.009163 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.009434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.012253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.013429 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.026556 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.048190 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:12.088760 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:09:12.183161 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.185588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.188585 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.189249 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.201507 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.213233 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:12.277669 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240503 02:09:12.328455 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.332126 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.333003 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.336519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.337824 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.341085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.430979 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.433555 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.436033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.442780 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.444289 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.474784 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:12.499732 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:09:12.531884 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.534046 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.535982 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.537187 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.538182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.543905 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.769968 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.776073 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.777338 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.777768 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.785884 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.790978 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.872392 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.872568 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.872859 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.877569 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.902566 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.904996 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:12.944380 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:12.944387 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.944576 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.955789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:12.957749 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:12.966084 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:13.025777 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240503 02:09:13.205405 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.207341 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.208349 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.215936 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:13.224310 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.227425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.291411 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.291929 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.292894 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.306620 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.309135 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.309741 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:13.356355 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:09:13.392838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.397356 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.397715 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.402995 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.404852 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.408213 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:13.440687 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240503 02:09:13.526778 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.527359 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.538976 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.539191 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.539618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.540764 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:13.589179 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240503 02:09:13.699890 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.700160 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.711661 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.712396 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:13.713209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:13.721105 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:09:13.721196 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.811705 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.812456 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.817553 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.825264 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:13.825577 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:13.825876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.901897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.905593 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.905663 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:13.905731 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:13.907001 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:13.935698 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.122795 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.122895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:14.124317 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.138470 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.138569 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.148025 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.247673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:14.253355 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.253710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.258630 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.259007 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.268174 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.268190 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.278564 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.285622 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:14.294571 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.301682 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.315341 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.612191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.616153 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.616206 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.618666 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.634083 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:14.634708 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240503 02:09:14.637478 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.660230 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.666474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.667431 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:14.670043 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.675868 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.683243 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.766875 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.767086 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.770506 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:14.772949 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.777201 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.778818 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.973356 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.976846 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.981264 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:14.981873 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:14.983912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:14.988603 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.079596 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.080229 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.084771 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.089525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.102957 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:15.103195 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:09:15.111956 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:15.138620 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:09:15.190711 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.191326 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.193467 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.193563 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.193950 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:15.205037 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240503 02:09:15.210397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.296243 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.298635 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.298933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.299628 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:15.300515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.300559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.507339 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.510923 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.511596 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.513042 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.513314 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.515367 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:15.622778 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.631428 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.632644 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.633064 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.640102 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:15.643025 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.648970 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:15.668413 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.672192 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.675680 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:15.691879 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.693394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:15.772975 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:09:15.989624 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.993044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:15.996636 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:15.997435 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.010095 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.012778 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.012967 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.018450 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.019935 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.025386 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.048214 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:16.052086 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:09:16.075593 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:16.078769 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240503 02:09:16.129539 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240503 02:09:16.139446 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.139731 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.141216 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.149246 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.152547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.157887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:16.315529 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:09:16.363462 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.365520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.367533 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.375020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.383531 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.385887 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.469807 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.472599 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.478024 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.478258 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.500452 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.505942 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.593849 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.594113 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.598973 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.605794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.605947 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.606853 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.692694 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.693307 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.694222 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.700042 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.707310 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:16.710479 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.904256 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:16.904546 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.904811 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.908978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:16.918215 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:16.918780 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.021168 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.028149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.029399 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.033308 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.035027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.039108 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.045151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.059881 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.060243 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.062148 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.081521 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.097118 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:17.231240 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:09:17.300958 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240503 02:09:17.382102 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.382865 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.397419 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.399127 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.404335 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.412324 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.413672 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.418673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.426038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.439127 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.447973 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.456636 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:17.471423 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:09:17.545091 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.548810 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.550933 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.556525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.566218 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.570326 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:17.726306 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240503 02:09:17.754482 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.756306 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.756369 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.765233 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.770244 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.776717 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.864675 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.871075 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.871150 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.871898 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.876042 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.893617 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:17.970598 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.973515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:17.976821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.976821 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:17.978623 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:17.984647 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.072361 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.074324 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.078392 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.078855 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.079206 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.088392 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.287947 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.290390 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.292218 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.297761 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.308074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.317241 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:18.343080 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:09:18.348400 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:18.397524 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:09:18.407330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.411747 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.422734 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.423120 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.430114 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.431612 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.433219 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.436630 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.451184 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.459751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.470273 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.481118 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:18.561972 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240503 02:09:18.732175 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240503 02:09:18.788620 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.794967 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.805877 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.809005 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.811398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.818661 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.831912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.846072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.847496 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.847674 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.849318 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.857708 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:18.949402 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.950083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.951925 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:18.953233 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:18.956108 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:18.960755 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:18.983276 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:09:19.161935 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.164790 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.165570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.170482 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.177544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.177919 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.260313 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.260854 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.266166 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.273130 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.273406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.294144 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.375425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.375506 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.377575 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.378839 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.384399 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.405547 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.476033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.479065 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.481146 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.482118 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.494832 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.497372 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:19.593647 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:09:19.690644 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.690948 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.696321 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.697376 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.701342 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.710964 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.802899 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.806192 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.809134 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:19.814500 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.815104 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:19.815701 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.836383 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.838029 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.841969 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:19.851663 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:19.863386 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:19.868628 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240503 02:09:19.874344 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.184911 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:20.185182 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:09:20.188302 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.189735 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:20.189828 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:09:20.191010 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:20.191392 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240503 02:09:20.196712 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.216925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.247924 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.249058 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.252908 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.255154 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.262737 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.270711 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.342666 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.343116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.347575 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.350718 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.350965 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.360002 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.563625 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.563953 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.569566 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.570339 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.576112 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.578912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.665725 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.668093 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.678805 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.680398 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.687813 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.697676 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.774464 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.775519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.781486 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.788694 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.789072 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.789359 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:20.848369 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:09:20.886288 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.887024 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:20.898265 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:20.901966 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:20.913554 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:20.921495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.096658 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.098570 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.099382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.099921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.113051 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.113792 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:21.203802 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:09:21.220923 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.221650 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.231503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.231724 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.232225 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.232884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.236934 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.238610 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.239464 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.244693 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:21.258023 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240503 02:09:21.269621 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.271613 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:21.339594 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240503 02:09:21.384263 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:09:21.592382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.594972 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.598191 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.602802 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.606985 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.614497 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.614734 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.619439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.628365 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.644634 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.644886 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.671903 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.728879 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.731205 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.734055 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.734395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.734742 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.744385 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.961410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.962137 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:21.967618 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:21.967959 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:21.980955 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:21.984877 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.067840 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.069509 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.071344 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.072017 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.072772 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.079622 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.169438 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.171129 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.174273 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.174602 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.184826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.186637 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.279840 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.280213 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.280557 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.294785 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.301594 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.302225 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:22.319550 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:09:22.470307 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240503 02:09:22.479759 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.484503 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.485985 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.487478 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:22.490294 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240503 02:09:22.505363 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.512566 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:22.586745 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:09:22.608429 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.611107 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.615229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.615594 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.619942 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.626823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.638691 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.660591 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.661186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.666679 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.668758 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.672845 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:22.957712 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.958148 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.976728 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:22.979208 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:22.981611 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:22.989967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.015847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.020215 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.021801 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.022047 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.033784 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.038069 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:23.070060 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:09:23.120671 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.121096 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.126803 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.126819 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.146811 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.148237 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.230294 6487 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:23.230432 6487 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:09:23.230947 6487 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:09:23.231407 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:09:23.232065 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:23.232151 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:23.232232 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:23.232353 6487 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:23.345635 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.347402 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.348541 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.349704 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.349871 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.349993 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.446774 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.447263 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.448978 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.450481 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.468555 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.474862 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.559458 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.563720 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.564782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.570436 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.576982 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.580250 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:23.653618 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240503 02:09:23.662194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.663942 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.666615 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.671319 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:23.680054 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.689931 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:23.692219 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240503 02:09:23.881522 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.882947 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:23.884533 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.889400 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:23.896044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:23.908299 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:23.950480 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:09:24.019345 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.022154 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.026391 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.027089 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.027179 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.032363 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.037635 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.051476 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.057662 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.082324 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.091737 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.096390 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.387185 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.393607 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.397450 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.401317 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.404515 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.427933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.427968 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.435036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.440105 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.457336 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.461241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.465178 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.548530 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.550840 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.559054 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.563349 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.573345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.573874 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.759019 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.759302 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.760561 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.769404 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.783664 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.785396 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:24.862823 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240503 02:09:24.871901 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.872009 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.874785 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:24.875600 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.882730 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.884189 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:24.978658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.979517 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:24.980835 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.982295 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:24.983585 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:24.997048 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240503 02:09:25.009871 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.082176 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.082348 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.082357 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.086820 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.094120 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.098253 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:25.113226 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:09:25.299823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.303120 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.303856 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.305928 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.313215 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.313591 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:25.413762 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:09:25.414691 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:09:25.415094 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.420439 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.421315 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.429391 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.443956 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.449409 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.453629 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.453884 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.455827 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.461541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.468499 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.475888 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.795177 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.798029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.804683 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.807976 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.811307 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.819557 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.834558 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.837710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.842514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.861148 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.876312 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.888713 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.965772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:25.967015 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.970788 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:25.971817 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:25.977684 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:25.983959 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:26.067842 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240503 02:09:26.100759 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240503 02:09:26.131618 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.140568 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.142751 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.143319 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.155354 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.157819 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.271742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.276482 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.281044 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.284227 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.286515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.288439 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.378142 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.378352 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.382230 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.390406 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.391541 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.409929 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.472882 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.475519 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.476253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.479472 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.488135 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.504220 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:26.619169 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:09:26.693303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.693413 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.700186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.700670 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.707921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.709892 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:26.748911 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:09:26.808214 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.813513 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.823606 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.824896 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.824935 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.835147 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.844460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:26.845265 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.853258 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:26.855517 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:26.859905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:26.878047 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.184054 10251 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:27.184188 10251 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:27.184710 10251 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
I20240503 02:09:27.185726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:27.186049 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:27.186127 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:09:27.186213 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:27.186570 10251 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:27.196101 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.200553 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:27.203151 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.211429 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.218662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.219125 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.222476 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:27.230391 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240503 02:09:27.231765 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:27.233742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.244642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:27.248941 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.347815 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:27.348349 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.352829 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.353196 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.355310 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:27.360570 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.544667 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.546350 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.546443 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.547915 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.559525 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:27.567498 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:27.592485 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:09:27.650754 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.658845 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.661100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.661100 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.662181 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:27.671008 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240503 02:09:27.676890 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:27.707046 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:09:27.758446 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.758860 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.759785 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.769084 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.770335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:27.777343 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:27.866245 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:27.867274 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.878427 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.879000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:27.879115 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:27.894361 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:28.005506 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:09:28.071080 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.071664 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.072793 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.076531 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.080935 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.086396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.185300 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.186064 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.189637 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.197929 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.208925 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.209566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.220348 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.245359 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.248075 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.248301 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.261921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.263679 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.525555 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.528261 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.532178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.535962 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.551848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.553930 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.602252 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.605259 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.618774 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.621268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.621702 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.630807 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.680050 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:28.681202 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.688954 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.690248 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.690923 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.691591 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:28.715736 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240503 02:09:28.722167 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240503 02:09:28.923128 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.927464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.929622 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:28.930446 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:28.937960 11361 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:28.938064 11361 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:28.939740 11361 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:09:28.939985 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:28.940667 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:28.942075 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:28.942534 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:28.942613 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:28.947857 11361 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:29.024583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.025588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.026446 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.030160 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.038527 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.058458 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.135749 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.136641 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.142508 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.148962 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.152487 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.165537 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.245980 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.249838 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.252444 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.253026 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.259581 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.260080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:29.264724 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:09:29.447611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.453812 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.454277 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.458714 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.462045 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.476864 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.573863 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.574615 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.577318 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.585044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.594539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.596880 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.598110 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.614511 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.618402 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.619596 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.631223 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.644490 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:29.726454 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240503 02:09:29.860546 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:09:29.943368 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.943702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:29.955040 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.964735 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.967381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.974849 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:29.976044 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.981642 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:29.985446 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.987650 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:29.993268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.016785 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.099293 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.099344 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.107142 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.108387 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.114696 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.123955 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:30.192711 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240503 02:09:30.307250 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.307722 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.308976 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.311115 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.311914 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.318608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.414498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.414746 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.415959 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.424742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.426306 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.428372 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.527117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.529280 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.529464 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.534721 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.541302 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:30.542882 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:09:30.543856 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:30.548018 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:09:30.620978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.623550 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.625128 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.626251 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.626304 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.636006 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.848809 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.850627 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.855826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:30.860337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.860734 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.862247 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.963863 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.967558 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:30.967996 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.980712 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:30.981081 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:30.982547 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.009555 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.014647 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.016868 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.019965 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.020259 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.037460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:31.040364 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240503 02:09:31.057200 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240503 02:09:31.351197 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.352264 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.356007 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.356865 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:31.359658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.366708 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.398026 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.400602 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:31.405301 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.408771 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.408819 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.423954 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.507072 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.509765 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.511219 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:31.514029 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.515220 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.529633 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.725971 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.729197 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.730690 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.738886 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:31.741964 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.752621 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:31.768896 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:09:31.828289 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.834729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:31.835400 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:31.842367 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:09:31.851022 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.852455 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.864778 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.947685 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.949915 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.951571 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:31.953848 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:31.958881 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:31.964403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.053436 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:32.053483 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240503 02:09:32.055068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.057936 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.058430 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.062248 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.087482 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.260807 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.264369 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.270274 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.271881 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.285784 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.303076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.377056 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.379057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.383311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.384742 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.411571 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.420471 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.426790 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.428135 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.434556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.441388 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.455381 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.469300 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:32.469415 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:09:32.678958 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240503 02:09:32.731992 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:09:32.752622 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.752712 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.754438 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.755854 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.771432 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.784258 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.820201 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.825646 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.825767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.826149 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.828331 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.839010 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:32.934302 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.936443 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.939453 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:32.939929 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:32.940479 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:32.948407 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:33.018318 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:09:33.089690 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.089901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.091221 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.095077 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.102875 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.110697 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.247653 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.249419 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.250114 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.251438 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.252918 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.260224 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.356678 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.358592 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.358898 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.361342 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.381290 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.381986 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.451659 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.457188 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.458699 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.459940 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.466941 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.478013 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:33.501619 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240503 02:09:33.655158 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240503 02:09:33.670022 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.671389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.671505 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.680516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.690366 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.691645 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.820021 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.823397 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.831004 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.834498 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.850880 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.851706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:33.871656 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.884321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.887004 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:33.896580 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:33.912475 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:33.922320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.185669 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.187211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.189939 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.193310 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.195253 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.209218 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.261957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.262823 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.269923 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.283840 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.285368 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.288362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:34.314065 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:09:34.345218 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.349095 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.349256 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.354087 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.355599 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.379503 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.592048 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.594970 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.596262 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.597666 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.598330 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.626343 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.692102 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.693509 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.695362 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.695883 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.702525 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.712810 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.794026 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.797870 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.798493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.800180 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.800442 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.801226 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:34.858856 13551 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:34.858999 13551 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:34.859457 13551 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:09:34.860602 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:34.860805 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:09:34.860870 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:34.860987 13551 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:09:34.877728 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
I20240503 02:09:34.917477 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:34.917477 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.920889 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.921458 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:34.921769 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:34.938621 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:34.992821 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240503 02:09:35.103062 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.106599 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.108549 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.109261 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.118039 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.118445 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:35.128862 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:09:35.245055 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.245605 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.248246 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.256305 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.261157 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.269868 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.282819 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.284395 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.291982 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.314642 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.315694 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.322855 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:35.470609 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:09:35.575424 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:09:35.615135 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.619704 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.627866 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.633369 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.635587 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.643455 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.649988 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.652230 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.654486 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.662688 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.672215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.699306 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.768677 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.772029 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.773456 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.782068 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.782801 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.785949 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.984411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:35.989769 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.990264 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:35.991374 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:35.992019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:35.992280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:36.035395 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240503 02:09:36.062554 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
I20240503 02:09:36.088251 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.088376 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.091408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.100921 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:36.101099 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.114811 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.194944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.195864 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.196390 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.196931 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.202466 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.215456 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:36.302392 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.302448 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.304962 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.305315 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.314220 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.337726 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:36.504913 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.509335 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.512722 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.516316 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.516619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.524421 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:36.610294 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:09:36.637449 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.641470 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.652304 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.659097 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.663939 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.672863 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:36.682250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.688606 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:36.695597 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.699293 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:36.704429 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:36.706713 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.017897 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.023749 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.028136 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.028244 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.029554 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.032171 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.032632 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.040280 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.051625 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.060367 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.064095 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.072422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:37.112533 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240503 02:09:37.168751 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.169798 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.170852 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.179116 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.184494 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.191206 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:37.296772 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240503 02:09:37.365229 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:09:37.366065 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:09:37.387840 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.389504 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.390149 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.390946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.391023 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.404639 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.491011 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.491914 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.494385 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.494642 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.501495 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.509583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.595106 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.596081 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.596381 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.598424 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.615954 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.623922 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.696065 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.698235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.699831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.704608 13981 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:37.704710 13981 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:37.705098 13981 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
I20240503 02:09:37.706192 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:37.706436 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:37.707227 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:37.707293 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:37.707556 13981 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:37.711719 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:37.729336 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:37.788803 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:09:37.906936 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.909018 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:37.909399 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.911183 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:37.912086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:37.912276 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:38.010708 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:38.023710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.028975 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.029495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.047868 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.047936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.054421 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.061300 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.067335 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.070178 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.078087 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.079180 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.092114 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.397626 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.400925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.406049 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.407363 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.423981 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.425823 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.451323 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.460677 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.481662 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.485891 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.486161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.488188 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.575428 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.576164 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.579397 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.580116 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.584769 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.585870 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:38.732592 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240503 02:09:38.734117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.736106 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.741325 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.745748 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.753180 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.755570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:38.780335 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240503 02:09:38.886844 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.887413 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.889019 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.895898 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:38.906610 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.912478 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:38.988924 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:38.992274 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.993734 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:38.996305 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.008816 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.009953 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:39.074545 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:09:39.085738 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.087261 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.089926 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.091692 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.096726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.097910 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:39.305683 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240503 02:09:39.306823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.307744 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.309249 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.315171 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.315392 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.319403 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.418661 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.425235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.425662 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.426697 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.434458 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.437109 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.447172 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.471077 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.473392 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.474381 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.481196 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.483927 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:39.776348 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240503 02:09:39.787164 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.790117 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.795912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.801726 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:39.804710 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.809417 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.835494 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.839208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.843057 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.850317 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.857071 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.867358 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:39.931249 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:09:39.959447 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.960919 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.961637 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:39.965200 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:39.972157 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:39.981178 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:40.092357 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:09:40.161091 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.161142 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.168812 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.169102 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.178961 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:40.184849 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.264886 13791 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:40.265014 13791 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:09:40.265545 13791 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
I20240503 02:09:40.268860 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:40.270718 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:09:40.272208 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:40.272938 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:09:40.273006 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:40.272995 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.273165 13791 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:40.275491 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.281216 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.290516 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.290786 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:40.372474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.372798 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.376415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.378211 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.383952 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.384312 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:40.461737 13981 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:40.461864 13981 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:40.462275 13981 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:09:40.464202 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:09:40.464376 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:40.464443 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:40.464579 13981 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:40.478801 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.480896 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.483063 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.485008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.494819 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.501762 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:40.633196 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:09:40.689000 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.693231 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.696832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.701031 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:40.701102 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.701658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.804445 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.810463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.812381 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.812871 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.823984 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.844241 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.850596 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:40.852545 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.857028 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:40.877532 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:40.880473 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:40.888746 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:40.918483 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240503 02:09:41.183007 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.183810 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.207389 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.210781 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.210848 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.241022 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.251323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.251669 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.264163 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.274963 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.286073 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.289045 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.370064 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.372901 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.378257 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.391014 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.391335 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.398749 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:41.434247 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240503 02:09:41.521797 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:09:41.578413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.580382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.582157 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:41.591212 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240503 02:09:41.591460 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.594000 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.610163 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.693243 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.695335 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.696713 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.697067 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.698807 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.708765 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.790267 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.797077 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.797370 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.798839 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.799795 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.811511 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.899614 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.904042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:41.904417 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:41.906826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:41.913859 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:41.927634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.108460 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.111575 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.111747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.112231 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.132484 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.133183 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:42.146622 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:09:42.236591 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.237361 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.240203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.243121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.253913 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.261801 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.262717 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.268224 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.280540 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:42.287911 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:42.288296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.308998 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:42.326134 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240503 02:09:42.490269 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:09:42.584060 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240503 02:09:42.599436 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.606210 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.609321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.610006 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.618105 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.621795 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:42.639516 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.652240 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.671738 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.674314 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:42.681167 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.689410 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.737735 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.738498 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.743249 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.747951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.750128 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:42.753358 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.981565 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.981946 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:42.985216 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:42.987195 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:42.996264 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.008859 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.077581 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.080750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.082864 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.090451 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:43.090981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.093034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.181695 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.182390 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.187633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.196193 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.198669 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.207278 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:43.231365 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:09:43.298399 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.298410 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.303948 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:43.311779 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.316910 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.322265 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:43.479177 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240503 02:09:43.497890 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.497933 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.499282 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:43.510310 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.510614 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.524503 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.619297 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.629860 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.630352 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:43.630940 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.642985 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.643397 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.651330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.660519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.666905 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:43.667560 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:43.675572 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:43.678153 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:43.849656 15960 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:43.849773 15960 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:43.850134 15960 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:09:43.850790 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:09:43.850937 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:43.850999 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:43.851128 15960 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:09:43.962186 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240503 02:09:44.000784 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.002120 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.007903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.015954 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.018260 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.019198 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.021610 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.025805 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.028298 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.031515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.031808 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.049438 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:44.065137 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:09:44.152799 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.154296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.157861 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.158421 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.165482 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.174026 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.355048 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.355446 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.355520 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.356338 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.364521 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.364921 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:44.410521 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:09:44.457520 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240503 02:09:44.469151 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.472085 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.474571 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.475966 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.488092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.492133 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.576747 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.583330 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.583974 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.588481 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.591694 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.599917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.681501 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.685498 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.688241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.689451 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.691592 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.708102 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:44.891232 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.891366 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.900523 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:44.908212 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:44.908797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:44.909688 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:44.954569 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:09:44.991027 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240503 02:09:45.011794 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.014122 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.022771 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.027652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.031944 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.037050 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.039331 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.049932 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.066802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.067160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.070776 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.071336 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.178750 16948 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:45.178874 16948 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:09:45.179286 16948 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:09:45.180586 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:45.180680 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:09:45.180761 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:45.180891 16948 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:45.360798 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.362331 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.363209 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.366189 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.372459 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.392958 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.415247 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.416914 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.419114 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.422255 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.425304 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.448359 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.526021 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.528579 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.529302 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.540766 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.541085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.541541 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.747210 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.747967 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.749933 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.750530 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.752472 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.761502 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:45.779383 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:45.851202 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.854439 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.855840 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.859961 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.865178 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:45.865630 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.955499 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.960188 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:45.960311 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.962004 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:45.976590 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:45.977063 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.061393 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.061518 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.067821 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.070161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.071821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.076737 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:46.257063 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240503 02:09:46.265478 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.267699 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.268772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.276808 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.277200 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:46.296583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.380949 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.388123 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.395303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.400453 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.402037 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.410853 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.412307 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.414415 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.417217 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:46.422008 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.423681 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.468552 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:46.611166 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240503 02:09:46.765270 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:09:46.767102 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.772804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.786162 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.787832 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.788558 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.800252 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:46.806857 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.816157 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.830792 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:46.838939 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.846189 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.847446 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:46.860927 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240503 02:09:46.936439 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.940866 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:46.941069 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:46.942397 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:46.950520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:46.951237 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.120998 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.123296 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.127491 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.128337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.143110 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.143440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:47.193137 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:09:47.242794 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.246728 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.248366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.249524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.252298 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.258425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.351155 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.352330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.354007 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.354602 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.358377 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.371146 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:47.425899 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240503 02:09:47.456362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.458709 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.462221 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.466195 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.468777 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.486182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:47.587247 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 02:09:47.625329 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:09:47.659332 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.661428 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.662364 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.662629 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.665454 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.665891 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.766680 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.774012 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.775588 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:47.778434 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.780012 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.787621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.839490 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.843662 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:47.844345 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.849938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:47.864053 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:47.869485 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.082653 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.086931 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.090863 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.091668 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.094460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:48.100239 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.198282 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.204849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:48.205372 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.208114 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.215240 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.243368 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.246490 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.248716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.250116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.256484 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.268180 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.290903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:48.405778 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:48.557705 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.564009 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:48.565207 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.566686 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.568885 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.571951 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.629375 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.633183 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.633843 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.640331 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.640465 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:48.640774 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240503 02:09:48.643309 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.726963 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.733768 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:48.733943 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.735193 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:48.744182 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.748221 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.874377 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.878520 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:48.880340 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.880584 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:48.897140 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:48.905647 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.037585 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.038205 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.052315 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.056807 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.058161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.063942 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.145821 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.149158 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.154208 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.161850 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.164672 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.165861 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.234656 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.237975 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.238221 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.244518 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.247267 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.247929 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:49.294334 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:09:49.380175 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240503 02:09:49.459796 15960 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:49.459947 15960 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:49.460376 15960 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:09:49.461035 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:09:49.462718 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:09:49.462792 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:49.463019 15960 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:49.471002 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.472318 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.474156 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.475113 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.476639 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.481901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:49.553265 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240503 02:09:49.575156 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.575878 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.583542 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.586444 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.587658 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.587921 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:49.625380 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:09:49.646733 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.646733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.649930 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.656045 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.656278 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.663975 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:49.899442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.903235 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.907109 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.911696 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.920749 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.932575 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:49.972151 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240503 02:09:49.995482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:49.997149 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.997423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:49.998172 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:49.999274 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.000942 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.097715 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.100548 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.105682 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:50.113754 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.115638 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.117019 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.221521 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.222849 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.223129 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.233835 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.234180 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.249146 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:50.316004 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:09:50.407174 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.409121 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.411430 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.413436 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.424244 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:50.424654 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.531721 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.532508 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.532871 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:50.551952 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.553936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.554903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.589757 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.591512 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.596374 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.614956 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:50.617581 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.617697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.743120 13791 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:09:50.743252 13791 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:09:50.743913 13791 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:09:50.744305 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:50.744305 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:09:50.745196 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:50.745271 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:09:50.745328 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:09:50.745455 13791 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:50.883994 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:50.887066 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240503 02:09:50.887460 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.889256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.889493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.904938 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:50.912256 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:50.962517 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.964138 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.969208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:50.972044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:50.990092 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.004434 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.045823 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.048765 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.051079 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.061933 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.065287 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.067065 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:51.234339 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240503 02:09:51.307108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.308476 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.308490 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.310510 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.313030 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.314195 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.394580 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.396576 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.399152 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.400818 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.404502 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.419365 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.493556 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.496519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.502480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.502745 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.509686 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.512964 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.622651 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.624675 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.626317 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.626847 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.627970 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.646694 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.815872 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.815894 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.820905 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.823976 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.832449 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.837050 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:51.883042 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:09:51.919420 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240503 02:09:51.941498 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.953574 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.954974 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.963733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.964967 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.972494 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:51.975697 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:51.976166 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:51.979020 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:51.979384 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.002877 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.022425 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:52.110296 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240503 02:09:52.149968 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:09:52.236878 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240503 02:09:52.305158 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.305979 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.306605 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.322407 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.324944 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.331538 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.372288 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.375530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.377028 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.379038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.379038 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.399113 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.483414 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.483979 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.484947 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.486943 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.487226 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.489116 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.620939 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.621440 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.627897 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.633158 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.634546 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.644235 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.793645 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.793658 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.798210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.807576 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.807960 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.808056 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:52.856240 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:09:52.912626 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.913398 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.915905 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.925479 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.931957 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.936965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.980619 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.982550 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:52.987036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:52.987267 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:52.990446 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:52.999878 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:53.045684 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:09:53.224617 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.225335 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.225608 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.235922 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:53.239464 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.240540 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.332620 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.341814 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.345424 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.355509 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:53.359697 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.377238 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.377684 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.384166 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.386520 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.397068 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.397492 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.415144 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:53.435606 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:09:53.710711 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240503 02:09:53.713196 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.714704 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.718173 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.724607 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.725730 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.730437 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.735934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.748992 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.751199 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.754796 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:53.756281 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:53.782238 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.870023 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.871346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.879431 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:53.880053 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:53.888046 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:53.900071 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.070880 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.071079 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.073304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.076083 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.077299 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.077787 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.178570 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.182651 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.185140 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.191918 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.192881 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.196935 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.281271 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.288165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.288620 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.289769 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.303529 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.312202 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:54.384292 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240503 02:09:54.387307 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.392452 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.394848 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.397840 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.411942 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.415586 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:54.512221 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240503 02:09:54.581678 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:09:54.600816 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.604925 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.611320 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.616673 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.617853 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.628322 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.722383 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.727452 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.738076 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.739290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:54.739805 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.741922 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:54.748633 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.750414 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:54.767714 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:09:54.773456 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.773561 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:54.788333 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:54.794517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:54.969065 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240503 02:09:55.080796 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.081636 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.083657 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.087908 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.090947 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.115926 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.142194 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.142921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.150449 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.165136 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.166328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.175621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.255864 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.256666 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.262981 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.264323 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.264727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.265425 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:55.286362 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:09:55.407507 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.408243 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.412631 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.418144 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.421690 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.436251 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.567122 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.570376 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.573583 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.580210 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.586382 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.600420 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.679039 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.683054 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.689811 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:55.705086 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.705960 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.709604 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.757364 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.758723 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.759621 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:55.765419 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:55.765594 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:55.770068 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:55.910600 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:56.006772 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.008335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.016366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.018050 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.018690 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.020385 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.118610 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.118885 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.127017 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.130312 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.148396 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.160313 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.169306 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.177318 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.178584 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.179049 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.184319 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.206339 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:56.360653 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240503 02:09:56.504019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.506014 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.515347 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.516605 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.518839 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.533916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.544308 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.554409 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.560894 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.570209 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.576700 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.585773 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:56.594552 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240503 02:09:56.665746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.666637 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.670136 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.673307 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.674324 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.688083 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.872578 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:56.873934 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.876614 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.882722 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.888160 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.903149 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:56.972462 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.977170 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:56.977607 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.977716 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:56.982209 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:56.999608 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240503 02:09:57.000319 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:09:57.000886 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.078963 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.080771 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.082739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.090342 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.090812 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.091401 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.188073 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.189445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.197453 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.197794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.208317 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.212535 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.391556 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.394726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.396049 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.396063 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.414638 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.420104 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:57.424998 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240503 02:09:57.429049 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:09:57.462549 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:09:57.515995 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.518668 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.522831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.534693 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.535513 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.537498 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.544350 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.546890 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.551937 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.561036 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.571426 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.594642 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.887027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.887050 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.891852 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.900842 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.913950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.917325 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.924738 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:57.926054 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.927618 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:57.935796 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:57.948135 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:57.957293 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.041604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.045050 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.052462 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.054179 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.064951 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.066632 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.247686 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.248370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.249622 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.251991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.261826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.272094 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.356101 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.356101 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.358392 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.360070 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.362434 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.378183 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:58.444634 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:09:58.461166 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.462038 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.462488 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.464956 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:58.480984 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:09:58.481215 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.488808 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.570983 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.571214 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.571647 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.573592 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.581312 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.581976 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:58.674238 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240503 02:09:58.785037 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.785516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.788038 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.788501 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.796389 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.801036 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.900923 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.908305 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.912290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:58.922595 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.925367 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.935207 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.942487 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.948158 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.951910 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:58.952181 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:58.960438 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:58.975879 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:09:59.045979 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240503 02:09:59.279915 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.281915 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.282990 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.291225 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:59.295912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.298772 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.299021 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.300034 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.301277 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.320319 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.330186 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:59.351531 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:09:59.397222 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240503 02:09:59.431465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.432894 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.432960 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.435464 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.439927 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.445176 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:59.490309 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:09:59.633673 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.635253 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.638406 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.645792 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:59.652069 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.655947 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.740583 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.743000 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.748956 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.751920 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.756235 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.767069 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:09:59.818207 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:09:59.852167 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.852833 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.856020 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:59.856892 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.865471 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:09:59.866561 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240503 02:09:59.872349 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.881595 18422 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:09:59.881708 18422 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:09:59.882161 18422 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:09:59.883273 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:09:59.883430 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
I20240503 02:09:59.883489 12711 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:09:59.883744 18422 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:09:59.964119 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.965117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:09:59.967561 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:09:59.968349 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:09:59.971680 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:09:59.979545 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.145248 13791 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:00.145359 13791 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:10:00.146080 13791 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:10:00.146579 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:10:00.147505 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:00.147727 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:00.147820 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:00.148058 13791 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:10:00.149560 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:10:00.179069 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.181664 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.186872 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.187052 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.193397 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.208694 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.291261 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.293700 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.300200 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.314457 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.326304 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.331122 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.332726 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.343888 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.357239 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.360203 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.361094 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.370635 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.638129 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.639065 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.646257 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.646667 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.655225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.676676 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.704169 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.707656 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.708160 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.713670 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.720396 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.723122 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.803972 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.805716 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.808826 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.810706 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.811947 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:00.812557 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.973232 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.974025 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.975451 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:00.985689 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:00.986105 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:00.996260 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:01.083575 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:01.116132 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.118211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.120437 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.125303 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:01.126130 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.140467 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:01.192481 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240503 02:10:01.217463 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:01.217510 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.219625 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.227655 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.228911 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.242410 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.320008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.324846 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.326185 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.327909 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:01.336490 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.336803 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.533999 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.534868 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.536340 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.537747 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.540397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.548115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:01.569499 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240503 02:10:01.653967 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.658661 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.659024 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:01.661517 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.663841 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.666121 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:01.680078 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.684791 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.700775 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:01.710461 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:01.712788 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:01.713160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:01.735635 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240503 02:10:01.903307 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:10:02.021190 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.026165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.030021 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.038364 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.041541 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.043954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.069717 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.069990 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.071254 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.074846 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.086323 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.089818 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.186072 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.188602 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.190881 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.190891 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.191610 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.215348 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.356896 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.360122 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.367363 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.369977 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.372285 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.388140 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.502422 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.507455 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.507928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.517542 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.528267 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.535738 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:02.556864 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:10:02.593489 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240503 02:10:02.612740 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.618850 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.618985 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.619333 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.620545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.633767 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:02.683068 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:10:02.706153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.706153 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.706758 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.713003 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.717741 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:02.719658 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.925952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:02.931649 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.931681 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:02.931916 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.933647 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:02.937036 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.036530 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.040091 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.052582 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.056226 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.061828 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.067083 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.067945 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.080137 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.083420 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.095654 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.096195 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.121021 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.409420 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.409821 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.411872 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.412457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.415508 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.419004 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.437383 20326 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:03.437525 20326 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:10:03.440204 20326 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:10:03.448330 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:10:03.468648 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:03.469355 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:03.469434 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:03.469681 20326 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:03.477964 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.484359 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.489023 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.489693 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.490440 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.491611 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:03.567409 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:03.568506 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:03.570955 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240503 02:10:03.571432 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.575722 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.582247 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.587920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.588153 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.798187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.801244 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.809212 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.813419 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.815848 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.825894 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:03.915055 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.916267 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:03.920495 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.923965 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:03.926134 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:03.932322 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.017093 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.021994 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.022104 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.024699 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.032481 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.041523 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.124575 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.128697 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.129159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.130189 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.133716 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.151181 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:04.326310 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240503 02:10:04.333101 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.339080 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.342710 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.345039 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.347299 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.353247 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:04.391366 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240503 02:10:04.453430 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.456698 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.464640 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.476548 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.477408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.480616 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.490464 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.508262 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.513511 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.514859 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.530282 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.537593 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:04.619232 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:10:04.848883 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.850843 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.851671 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.854105 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.862706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.875985 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.876279 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.889236 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.897574 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:04.906213 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:04.912597 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:04.929045 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.991035 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.997493 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:04.997539 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240503 02:10:04.997797 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.998206 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:04.998628 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.010332 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:05.041824 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:10:05.216646 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.216719 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.218340 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.221698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.222366 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.243932 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.312951 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.313123 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.313364 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.319993 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.331048 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.340255 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.414884 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.418843 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.421958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.428828 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.431237 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.437912 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:05.484089 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:10:05.533577 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.539368 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.540267 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.540359 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.540735 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.561744 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.727185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.728500 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.734236 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.739485 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.747905 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.749156 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.850240 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.860625 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.860999 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.865564 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:05.876286 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.888762 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.899117 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.912683 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.913900 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:05.915608 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:05.919981 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:05.941084 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.194360 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.197571 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.200488 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:06.203034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.206718 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.208038 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:06.280311 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240503 02:10:06.288139 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.301373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.302457 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.302755 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:06.314642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:06.319831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.359383 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.363466 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.367347 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.369037 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.376310 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:06.383280 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:06.395884 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:06.620615 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.620797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.624344 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.627548 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:06.635255 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.639608 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:06.716284 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.716946 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.718482 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.718544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.728935 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:06.737128 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:06.824859 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.825907 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.835920 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.837352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.845292 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:06.846838 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:06.934863 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240503 02:10:06.941164 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.942605 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.943135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:06.944047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:06.945299 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:06.963096 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:06.966428 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240503 02:10:07.091848 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:10:07.163524 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.167125 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.168747 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.169834 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.172341 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.191479 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:07.264137 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240503 02:10:07.270078 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.273468 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.280198 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.283730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:07.301826 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.313668 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.342427 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.343961 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.344636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.346616 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.352850 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.363910 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:07.453006 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:10:07.627338 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.627648 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.628296 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:07.631062 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.634719 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.648044 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.700582 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.702329 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:07.702322 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.702766 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.704114 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.707892 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.797744 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.801725 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:07.802742 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.805856 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:07.807884 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.819375 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.937598 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:07.938169 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:07.948976 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:10:07.949929 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:07.951429 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.952745 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:07.960979 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.110519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.116845 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.117763 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:08.118036 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.123445 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.140096 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.225163 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.227739 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.230043 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.237133 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:08.242655 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.248392 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.303961 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.307533 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.309063 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.309280 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.309540 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.336892 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:08.542285 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.542990 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.543370 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.552271 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.556018 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:08.567543 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:08.627590 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:10:08.645004 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.647711 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.653214 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.653925 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.671087 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.692826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:08.699535 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.699600 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.701565 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:08.702255 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:08.711201 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:08.717371 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:08.776734 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240503 02:10:08.925007 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:10:08.998350 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.001423 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.003180 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.012465 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.013923 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:09.018697 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.063774 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.070152 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.073915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.076905 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.079212 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.079440 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:09.176115 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.176189 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.178920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.179152 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.190325 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.191700 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:09.339977 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.343001 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:09.343267 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.345507 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.347894 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.362185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.483258 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.486932 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.488266 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.488313 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:09.488991 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.503187 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:09.504623 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240503 02:10:09.591120 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.597883 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.598529 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:09.599263 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240503 02:10:09.603399 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.613140 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.626176 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.680915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.681455 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.687215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.690232 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.691069 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.704841 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:09.754009 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240503 02:10:09.763357 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:10:09.911098 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.911439 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.913674 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:09.914562 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:09.926780 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:09.939718 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.013018 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.015718 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.024094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.043004 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.043512 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:10.044373 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.066833 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.076320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.082837 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.085517 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.087483 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:10.093886 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:10:10.098090 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.382794 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.384719 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.398011 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.398905 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.403923 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:10.411087 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:10:10.417604 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.442520 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.443636 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.448153 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.448771 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:10.459223 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.465760 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.549664 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.552721 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.553920 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.554729 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.555274 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.566555 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:10.716915 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.719375 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.722975 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.729543 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.735078 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:10.735805 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.767239 13791 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:10.767381 13791 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:10:10.767798 13791 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:10:10.768275 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:10:10.769161 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:10.769255 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:10.769340 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:10.769487 13791 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:10.865625 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.866881 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.867568 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.874852 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:10.875553 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.882159 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.973811 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.974526 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:10.976238 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.986909 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:10.990978 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:10.992588 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.066849 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.069809 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.070394 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.076009 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.082669 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.098192 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.283290 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.284526 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.290496 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.291916 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.299672 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.300318 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.409339 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.410998 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.411408 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.421557 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.421741 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.422536 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.422750 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.439010 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.451462 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:11.464200 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:11.465162 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.468468 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.477854 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:11.588883 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240503 02:10:11.772732 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.781929 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.787031 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.789570 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.796744 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.806736 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.816181 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.836409 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.837559 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:11.848678 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.853627 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.860867 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:11.939219 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240503 02:10:11.952589 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:11.953747 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.953802 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.955729 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:11.958268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:11.968135 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.165850 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.169173 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.170446 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.173820 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.175885 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.187922 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:12.207051 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:10:12.274650 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.279500 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.281543 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.281754 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:12.285784 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.297288 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:12.308298 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240503 02:10:12.333627 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240503 02:10:12.379366 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.382809 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.385300 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.386251 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.388581 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.396103 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:12.484220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.485291 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.487210 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.500774 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.511070 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:12.512557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:12.567718 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:10:12.690394 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.691795 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.695482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.695928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.696529 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.698369 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:12.804018 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.811316 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.812191 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.813640 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.819939 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.831115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.833221 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:12.838999 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:12.840425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.847424 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:12.853566 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:12.878151 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:13.096748 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:10:13.179912 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.187433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.187664 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.187737 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.194909 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.201020 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.240229 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.240613 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.240700 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.242493 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.243299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.264238 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.345207 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.348119 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.353786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.358350 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.361019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.361507 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.492566 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.493628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.493702 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.508916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.516618 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.522994 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.657493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.657537 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.662308 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.664261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.665217 24835 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:10:13.665320 24835 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:10:13.665709 24835 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:10:13.666452 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:10:13.667143 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:10:13.669098 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:13.669181 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:10:13.669484 24835 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:13.674422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.680862 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.769263 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.772781 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.773339 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.783732 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.783955 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:13.809938 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.862474 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.863708 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.865902 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:13.872566 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:13.873423 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:13.879832 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.107036 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.107439 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.115573 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.116487 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.134687 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.134886 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:14.138597 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:10:14.220531 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.225508 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.237126 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.243695 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.252370 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.259488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:14.264202 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240503 02:10:14.265002 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.265352 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:14.269598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.284057 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.291116 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.301381 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:14.470202 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:10:14.470235 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240503 02:10:14.595551 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.596197 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.598246 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.604727 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:14.624317 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.629994 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.635524 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.647150 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.655974 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.659933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.664327 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:14.670341 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:14.686291 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240503 02:10:14.758891 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.760044 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.767619 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.769979 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.770871 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:14.771301 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.948639 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.949486 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:14.952702 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:14.962350 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:14.964277 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:14.968752 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:15.007907 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240503 02:10:15.023509 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:10:15.073207 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.073421 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.073588 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.076758 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.077632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.089159 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.173060 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.174330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.176332 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.178644 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.188834 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.204138 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.272337 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.275523 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.275964 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.279901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.293085 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.307154 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.491088 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:15.491931 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:10:15.497875 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.504626 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.505229 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.505574 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.511901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.622723 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.626685 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.629333 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.629377 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.638345 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.648252 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.652330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.664358 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.673954 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:15.677778 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:15.682447 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:15.685210 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.995687 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.997817 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:15.998299 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.001353 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.007293 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.016525 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.028478 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.045436 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.046255 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.048403 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.049799 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.052866 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.149132 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.151686 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.160789 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.162106 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.170188 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.188657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.343689 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.344229 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.346189 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.347071 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.360685 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.360807 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.469054 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.474634 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.479597 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.480553 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.480955 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.482169 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:16.540172 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:10:16.576632 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.577730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.581836 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.590116 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.590384 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.592048 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.670271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.672013 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.673492 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.674839 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.677243 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:16.686301 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:16.825927 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240503 02:10:16.906001 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:16.907814 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.913095 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.919266 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:16.920272 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:16.932683 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:16.960554 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240503 02:10:16.997102 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:10:17.013501 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.014227 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.015240 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.019466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.037081 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.050545 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:17.058645 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240503 02:10:17.068867 25496 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:17.069006 25496 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:10:17.069432 25496 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:10:17.071981 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:17.072173 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:17.072238 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:17.072396 25496 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:17.089309 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.091949 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.095218 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.099216 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.106999 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.116096 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:17.221532 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240503 02:10:17.354893 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.358579 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.364225 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.380443 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.385427 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.388597 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.432902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.435040 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.438088 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.438659 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.442330 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:17.450167 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:10:17.470110 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.549474 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.550637 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.551744 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.555079 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.559954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.567255 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.676146 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.676988 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.681591 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.681811 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.686443 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.696445 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.862638 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.868117 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.868806 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:17.870393 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.871289 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.888334 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:17.981202 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.981528 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:17.983981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:17.990213 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:17.992874 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:10:17.999922 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:18.011539 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.040071 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.045702 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.055429 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.056542 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:18.058244 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.068832 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.325019 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.327294 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.328425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.340756 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.343575 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.363730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:18.380195 12621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:10:18.381668 12621 catalog_manager.cc:4573] TS 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:18.405990 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.410038 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.417193 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.425607 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.429947 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:18.436734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.490650 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.491309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.495106 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.500967 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:18.509418 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.512398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.725714 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.727648 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.728399 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.731392 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.733183 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.734335 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:18.786916 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:18.826925 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.829411 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.838419 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.848867 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.850950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.857179 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:18.940213 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.942487 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.943706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:18.943902 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:18.956454 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:18.958441 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:19.038450 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.040256 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.047644 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.054971 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.064118 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.066054 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:19.237116 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240503 02:10:19.250006 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.250499 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.254614 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.254712 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.258172 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:19.258539 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.371305 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.375203 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.378432 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.387385 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.387902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.388160 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.404917 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.416251 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.420653 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.420903 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:19.427851 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:10:19.441462 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:19.443461 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:19.560792 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240503 02:10:19.603379 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 097947dce87b409ca4a57cb752fdd5e2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:10:19.607254 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:10:19.693225 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240503 02:10:19.695609 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240503 02:10:19.747550 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.751401 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.757546 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.759675 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.769110 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.777616 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.780225 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:19.783161 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.792737 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.803934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.804179 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.824375 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:19.909451 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.910626 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.914994 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:19.917260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:19.920801 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:19.921237 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:20.012943 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:10:20.109108 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.114025 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.116887 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.118530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.118952 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.132525 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.229534 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.230353 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.234625 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.235070 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.244252 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.255842 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.339993 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.343977 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.345238 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.351941 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.355325 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.370303 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.443584 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.445235 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.446939 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.447311 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.454236 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:20.454581 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:10:20.480901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.655984 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.657889 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.658541 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.660951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.674990 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.686488 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.774308 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.779083 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.780215 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.786567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.789803 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.807931 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.818423 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.818827 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:20.819597 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:20.822633 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:20.843804 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:20.858594 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.146838 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.152525 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.157384 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.159652 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.164733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.165139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:21.189236 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.200598 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.203538 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.203572 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.207922 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.211827 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:21.307811 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.308810 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.311488 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.313797 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:21.321446 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.321640 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.525015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.525094 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.525553 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.525717 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.529225 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.538111 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:21.622659 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.623098 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.628580 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.635916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.637303 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.641836 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:21.644891 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240503 02:10:21.720211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.724412 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.726895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.729074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:21.730221 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.743782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:21.751242 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:21.835001 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.837054 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.840101 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:21.842816 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:21.856235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:21.856810 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:21.893949 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:10:22.028612 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.029351 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.034835 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.035904 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.037611 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:22.060436 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:22.125918 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240503 02:10:22.145190 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.154950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.157696 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.157713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.161764 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.169234 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:22.210917 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.211495 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.212608 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.217702 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.225975 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:22.236629 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240503 02:10:22.237397 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:22.434489 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240503 02:10:22.485651 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.490567 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.501849 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.505280 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.518311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.528075 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:22.528637 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:10:22.575621 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.580617 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.581619 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.585124 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.591789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.601703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:22.650344 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.655540 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.655704 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.658525 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.661952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.664440 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:22.884670 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:10:22.904750 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.905730 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.908957 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:22.909127 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:22.915365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:22.918252 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.002959 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.004590 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.006276 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.007460 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.014643 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.026957 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.112663 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.113312 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.116966 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.117730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.118320 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.136065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.217005 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.220988 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.223934 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.227937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.234215 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.249945 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.421002 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.424392 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.425426 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.428884 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.437296 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.444797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.553515 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.563565 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.564612 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.564787 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.566726 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.567919 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.569353 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.576421 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.581226 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.588655 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.591079 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.620492 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:23.926548 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:23.927919 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.933048 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.941138 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:23.941895 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.944300 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:23.951109 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.959332 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.962373 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.980139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.980676 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:23.984766 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:23.986167 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:24.060537 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240503 02:10:24.081842 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.084968 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.090498 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:24.092612 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.095285 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.103916 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.282178 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.287086 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.288831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.302657 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.304658 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.312675 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:24.400086 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.405691 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.408557 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.409217 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:24.412950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.424320 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:24.451747 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240503 02:10:24.519073 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:10:24.520211 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.523309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.532114 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.535992 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.547534 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.554747 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:24.612609 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.613744 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.618829 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.624270 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.628008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.630070 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:24.793699 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240503 02:10:24.851666 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.855156 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.856137 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:24.856954 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.859966 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.864182 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:24.902786 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:10:24.903656 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:10:24.952262 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.953698 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.961779 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:24.963156 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:24.977964 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:24.978235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.020766 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.024567 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.027500 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.028862 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.033531 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.035077 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:25.048957 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240503 02:10:25.286074 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.288348 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.295943 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.300065 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.300405 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.313688 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.373430 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.375169 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.376842 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.380738 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:25.381332 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:10:25.385237 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.391638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.474691 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.477944 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.483243 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.491281 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.491518 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.510737 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.601361 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.605204 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.606657 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.609443 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.611776 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.640877 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.793751 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.792661 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.794471 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.799830 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.806422 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.820118 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.903005 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.904584 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.909181 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.920361 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.924063 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.936064 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:25.989480 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:25.993489 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:25.993986 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:25.994932 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.003948 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.019651 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:26.234519 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.235913 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.241938 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:26.242165 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.243422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.270756 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.344537 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.344921 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.346166 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.361651 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.365474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.371625 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:26.401147 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.406129 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.408819 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.411595 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.412195 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.412750 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:26.635975 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240503 02:10:26.672262 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.679615 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.680151 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.682101 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:26.688750 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.693367 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.749305 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.752560 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.755388 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.756191 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:26.761473 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240503 02:10:26.772780 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.775652 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:26.812242 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:10:26.855618 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.857426 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.857616 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.859089 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:26.864359 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:26.878650 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:26.996166 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:26.996416 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.002599 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.005795 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:27.009896 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.010486 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:27.088397 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:10:27.170228 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.175523 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.177675 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.178226 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.183954 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.188362 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:27.248533 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240503 02:10:27.275671 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.281857 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:27.292681 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.293514 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.294054 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.303304 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.325536 26364 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:27.325659 26364 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:10:27.326128 26364 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:10:27.326550 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:10:27.327262 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:27.328138 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:27.328202 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:27.328382 26364 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:27.366070 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.366919 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.367170 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.372663 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.381366 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.392117 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:27.402830 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:10:27.485877 27239 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:10:27.486007 27239 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:10:27.486532 27239 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:10:27.487223 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:10:27.487506 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:27.487568 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:10:27.487686 27239 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:10:27.561496 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240503 02:10:27.608819 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.612484 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.614835 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.618523 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:27.618774 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.628978 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.723218 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.730322 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.731092 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.737342 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:27.743960 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:10:27.753857 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.753971 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.766731 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.773301 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.781858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:27.788203 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:27.794970 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:27.795464 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.100797 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.105154 13016 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.105369 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.108794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.119700 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.120040 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.129554 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.134912 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.142019 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.146703 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.156275 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.178673 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.256520 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.256834 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.258533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.263271 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.263657 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.268874 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.456805 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.469920 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.470204 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.470820 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.471514 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.490360 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.577418 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.578020 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.578282 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.585636 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.588357 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.588515 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.675936 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.676410 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.676849 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.682211 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.686359 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.702416 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.777791 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.781474 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.782672 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:28.787019 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.788828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:28.793516 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:28.960800 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:28.987951 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.992368 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.993681 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:28.994479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:28.996604 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:29.010075 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.109268 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.109797 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.115497 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.120493 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.124315 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.130334 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.132061 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.133917 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.144778 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:29.145247 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240503 02:10:29.152863 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.173573 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.178359 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:29.402025 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240503 02:10:29.485415 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.486562 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:29.492025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.493984 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.496104 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.506886 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.532480 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.535459 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.547660 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.557998 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.558404 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.559697 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:29.567842 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:10:29.615482 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240503 02:10:29.620272 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:10:29.646246 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.646848 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.648313 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.651190 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:29.651301 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.662760 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.813493 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.815232 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:29.818094 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.819020 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.821846 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.823941 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:29.956045 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:29.957350 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.956660 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:29.960383 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.960557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:29.971107 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.063042 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.068600 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.069778 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.072898 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.074173 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.077309 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:30.114629 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:10:30.172993 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.174764 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.177958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.180812 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.182005 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.203549 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:30.237915 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240503 02:10:30.382890 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:10:30.387181 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.387653 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.387857 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.388998 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.390372 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.391072 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.502753 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.509853 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.510684 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.514514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.525605 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.528808 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.551399 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.561360 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.563005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.565837 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.565905 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.578627 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.840796 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.850097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.856047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.859823 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.862941 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.891100 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.891896 27239 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:10:30.892001 27239 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:10:30.892400 27239 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:10:30.893196 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:10:30.893515 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:30.893589 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:10:30.893868 27239 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:30.928283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.929544 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:30.929912 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:30.930593 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:30.935542 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:30.938970 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.027542 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.030645 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.031208 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.031564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.032002 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.034826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.189770 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.189934 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.192799 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.195062 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.199928 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.219692 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.340196 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.342975 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.345132 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.346155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.347388 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.354027 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.442987 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.447139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.449386 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.451890 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.454890 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.470873 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:31.549593 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240503 02:10:31.553146 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.553414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.559818 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.566155 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.567132 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.572598 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.758472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.760717 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.763917 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.767580 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.767864 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.775547 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:31.849757 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:10:31.865077 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240503 02:10:31.868592 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:10:31.882452 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.883175 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.892011 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.893522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.893904 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.898427 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:31.900395 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.919943 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.926294 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:31.926406 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:31.944160 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:31.952629 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:32.181171 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:10:32.255471 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.257254 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.267005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.269632 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.273407 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.279075 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.286967 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.312005 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.314745 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.320655 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.323025 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.323523 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:32.386149 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240503 02:10:32.407231 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.410259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.412514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.424317 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.424611 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.426092 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.626793 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.630447 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.631834 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.636641 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.640082 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.656627 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.724232 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.726119 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.730718 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.734290 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.741711 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.743446 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.830142 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.835912 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:32.837009 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:10:32.838404 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.838654 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.843333 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:32.852458 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240503 02:10:32.861660 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:32.946533 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.946830 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.947849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:32.948415 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:32.950212 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:32.968675 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.148734 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.150869 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.152077 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.157390 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.164044 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.176518 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.273619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.277982 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.284924 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.286338 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.301683 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.303172 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.306270 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.319077 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.326431 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.348625 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.355082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.356103 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.631065 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.632905 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.635666 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.648541 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.648653 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.648816 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.700577 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.701336 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.701447 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.706051 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.712679 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.732216 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.808290 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.808789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.811921 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.812523 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.815238 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.816422 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:33.930692 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240503 02:10:33.963176 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.963397 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:33.964077 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:33.965263 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:33.969405 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:33.969902 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:34.011333 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:10:34.129741 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.130867 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.134582 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.142598 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.149394 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.156159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.243275 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.243480 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.251617 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.260270 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.267802 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.278879 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.322161 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:34.324594 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240503 02:10:34.325305 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.326103 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.332000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.342522 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.344851 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:34.430483 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:10:34.457850 27239 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:10:34.457993 27239 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:10:34.458401 27239 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:10:34.459079 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:10:34.459527 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:34.459611 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:10:34.459779 27239 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:34.571275 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.571573 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.572134 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.572652 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.576249 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.592444 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.652456 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.654466 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.664271 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.673635 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.675941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.678242 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.734730 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.738121 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.741048 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.741398 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:34.748610 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.760897 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:34.806182 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:10:34.916273 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240503 02:10:34.980175 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.980446 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.984400 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:34.992115 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:34.992426 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:34.993494 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.077226 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.080789 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.083072 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.083276 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.087949 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:35.105008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.181634 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.184408 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.188380 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:35.189749 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.196161 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.196616 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:35.285369 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:10:35.290822 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.294080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.299891 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.302930 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.303797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.304826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:35.478374 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:10:35.492298 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.493466 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.496208 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.500460 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.508018 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.527906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:35.529973 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240503 02:10:35.612895 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.624190 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.626611 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.629562 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:35.638245 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.639286 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.642235 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:35.647903 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.658553 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.677197 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.681082 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.687140 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:35.976089 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:35.983000 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.982997 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.984709 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:35.997000 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.011271 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.046569 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.048645 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.052528 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.059916 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.067931 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.069931 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.157955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.161407 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.162273 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.168903 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.174645 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.181222 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.303395 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.305110 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.309144 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.311939 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.318596 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.329327 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:36.433056 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240503 02:10:36.466060 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.471335 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.474150 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.479650 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.484695 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.506283 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.591089 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.592356 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.594990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.598099 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:36.601619 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.618031 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.665077 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.665192 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.666358 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.675019 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:36.681492 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.683878 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:36.805486 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:10:36.911046 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.916680 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.916841 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:36.921017 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:10:36.922574 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:36.923921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:36.924018 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:36.997197 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240503 02:10:37.029349 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.041363 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.041563 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.055501 32126 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:37.055613 32126 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:10:37.056185 32126 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:10:37.058972 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:37.060513 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.062775 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.065935 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:37.076268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.086606 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.090212 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:37.102985 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.116319 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:37.128559 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:37.128647 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:37.128921 32126 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:37.141320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:37.278566 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:10:37.412776 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.416200 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.417511 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.423326 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.424685 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.432049 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:37.443652 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240503 02:10:37.488052 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.489780 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.490307 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.490347 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.490639 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.519466 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:37.604425 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.605222 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.617089 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.618088 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.619455 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:37.619689 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.734185 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.735301 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.743909 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.744160 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:37.748139 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.757288 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:37.770778 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:10:37.919499 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.924425 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.925231 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:37.927054 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:37.932085 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:37.948066 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.030433 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.032760 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.038703 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.039005 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.044479 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.073801 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.109155 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.110327 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.110642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.110692 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.112746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:38.129699 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240503 02:10:38.134821 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.366322 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.366683 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.368700 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.369892 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.373325 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.381074 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.467964 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.477965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.478411 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.489926 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.493156 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.510378 13148 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.510612 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.513360 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.516283 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.520507 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.525955 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.537144 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.706058 31129 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:38.706204 31129 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:10:38.706635 31129 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:10:38.707172 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:10:38.707942 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:10:38.708114 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:38.708176 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:38.708319 31129 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:10:38.766212 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240503 02:10:38.858976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.864634 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.866240 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.867271 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.874974 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.876188 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.876796 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.878521 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:38.878788 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:38.882118 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:38.923518 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:38.938807 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.007841 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.008683 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.010648 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.012557 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.025964 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.028338 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.223686 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.229888 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.229934 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.235951 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.237272 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:39.244556 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:10:39.245309 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:10:39.246605 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.327332 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.328786 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.329659 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.332556 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.340492 807 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:39.340592 807 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:10:39.340984 807 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:10:39.342201 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:39.342950 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:39.343279 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:39.343348 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:39.343540 807 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:39.345151 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.422560 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.425173 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.426187 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.433537 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.444499 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.448160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.539530 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.540195 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.541611 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.550272 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:39.556887 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240503 02:10:39.562551 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.563411 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.739923 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.743067 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.743093 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.743947 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.745792 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.761900 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:39.830593 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:10:39.865069 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.868933 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.871879 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.872929 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.880489 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.884253 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.884343 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.888571 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:39.904433 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:39.917263 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:39.919636 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:39.934065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:40.157186 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240503 02:10:40.172840 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:10:40.245429 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.246287 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.248178 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.251204 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.259871 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.264837 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.265789 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.274186 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.276196 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.304072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.307883 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.309963 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:40.328292 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240503 02:10:40.389493 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.392345 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.394189 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.401031 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.404345 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.412428 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:40.536305 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240503 02:10:40.594509 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.594960 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.597478 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.599402 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.616196 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.617504 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.697654 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.698858 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.699724 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.699945 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.700557 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.709022 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.793622 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.797034 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.798518 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.798609 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.812105 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.818641 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.901325 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.901952 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:40.914289 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:40.916761 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:40.917284 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:40.918639 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.107335 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.108621 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.110333 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.118815 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.131119 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.136763 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:41.228853 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240503 02:10:41.232509 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.238330 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.238379 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.244097 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.246505 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.248086 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.251904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.256424 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.257609 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.264844 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.291960 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.302413 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:41.589673 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:10:41.612315 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.618799 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.619952 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.620236 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.620707 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.625306 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.630312 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.639019 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.651324 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.659034 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.668793 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.670217 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.756500 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.758268 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.759893 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.759955 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.772042 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.779439 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:41.958864 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.962579 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.962858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:41.964455 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:41.971096 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:41.989329 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:42.068763 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.070482 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:42.072607 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240503 02:10:42.073622 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.074527 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.076287 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:42.080664 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:42.160126 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:10:42.173190 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.174762 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.177374 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.179993 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.202459 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:42.210898 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.275071 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:42.279311 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.279381 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.280520 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.284145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.301928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:42.429167 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240503 02:10:42.492280 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.494344 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.505833 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.506415 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.512516 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.522730 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:42.559441 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:10:42.607838 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.610458 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.612718 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.620330 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.621266 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.633607 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:42.635450 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:10:42.665704 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.668728 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.669984 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.671154 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.671820 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.672541 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:42.940697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.946077 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.946718 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:42.947168 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:42.948683 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:42.961601 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.017679 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.020628 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.021250 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.031929 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.032594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.036346 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.095449 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.097427 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.100744 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.100785 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.109640 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.109906 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:43.162856 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240503 02:10:43.339285 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.341956 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.346493 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.349808 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.353713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.354383 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.427213 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.431211 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.433519 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.435945 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.436540 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.439067 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.537797 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.539746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.543643 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.543793 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.545694 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.547715 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.651659 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.653285 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.656534 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.659965 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.668974 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:43.673857 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.855082 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.859112 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.862612 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.864594 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.869163 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:43.874758 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:43.881453 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240503 02:10:43.983007 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.983317 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:43.992308 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:43.993124 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.003546 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.011502 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.011749 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.025544 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.026947 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.035434 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.040216 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.061043 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:44.158957 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:10:44.345111 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.348186 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.352123 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.354377 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.362897 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.377521 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.407804 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.408282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.408762 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.412159 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.424604 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.425730 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:44.471812 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:10:44.491904 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.493162 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.494288 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.496201 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.496204 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.508395 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:44.631870 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240503 02:10:44.721038 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.723839 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.723915 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.725261 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.737180 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.753743 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.816524 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.816788 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.820873 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.837255 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.841547 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.847889 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:44.923727 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.926472 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:44.930758 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:44.936086 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:44.946372 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240503 02:10:44.953603 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:44.958074 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:44.989660 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:10:44.997969 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:10:45.039501 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.043435 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:45.044369 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.045722 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.051929 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.052142 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.241919 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.242396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:45.246374 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.252496 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.253260 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.254084 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.346123 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.351928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:45.358790 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.358858 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.359169 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.359524 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.415508 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.415851 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.419014 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.420869 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.428439 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:45.441322 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:45.569571 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:10:45.668337 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.673007 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:45.676406 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.677000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.677640 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.685528 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.779970 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.782117 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.786466 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:45.789952 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240503 02:10:45.795951 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.798473 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.809957 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.810185 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.822808 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:45.823948 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.852102 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:45.858244 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:45.860702 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.020107 27239 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:10:46.020241 27239 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:10:46.020684 27239 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:10:46.021407 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:10:46.021715 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:46.021781 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:10:46.021911 27239 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:46.126816 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.127826 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.133816 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.135994 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.137077 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.138002 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.200474 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.204387 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.204856 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.208376 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.209985 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.221626 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.304548 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.304860 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.305922 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.313515 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.319015 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.320139 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.441390 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.442095 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.448294 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.450263 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.452337 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.456583 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:46.587917 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
I20240503 02:10:46.611661 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.611920 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.614908 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.621870 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.623669 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.631703 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.718964 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.719661 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.720070 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.740216 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.742187 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:46.748128 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.796191 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.796789 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.801647 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:46.803644 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:46.810295 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:46.817452 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:46.832114 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:10:47.040179 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.041188 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.045943 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.046286 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.049978 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.067421 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.170828 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.171730 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.173594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.183178 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.185468 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.200664 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:47.224608 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240503 02:10:47.233392 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.233462 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.239823 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.242484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.242539 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.252658 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:47.302045 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:10:47.309528 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240503 02:10:47.523886 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.529803 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.531296 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.533378 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.534873 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.537101 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.602484 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.604111 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.604542 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.612979 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.616243 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.628160 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:47.676710 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:10:47.700405 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:10:47.705072 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.708139 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.708408 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.711912 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.714305 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:47.732904 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.849165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:47.849771 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:47.850706 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.851969 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.856540 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:47.884685 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.017422 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.019254 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.020476 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.021556 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.031657 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.035544 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.119714 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.120888 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.122992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.123706 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.131192 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.144825 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:48.179821 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20240503 02:10:48.214116 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.216023 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.218302 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.220475 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.222162 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.244748 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.447109 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.449561 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.449906 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.450006 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.450534 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.466305 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.560580 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.563457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.569339 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.576000 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.579811 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.607744 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.609922 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.613090 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.613993 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.616053 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.620738 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.640914 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:48.922910 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.934679 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.936882 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.940716 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.941053 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:48.947196 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:48.955649 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240503 02:10:48.988605 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:48.995128 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:48.996520 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.001058 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.002005 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.022832 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.102046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:49.103338 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.109694 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.109859 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.110008 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.132866 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.251878 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:49.253376 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.253526 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.261550 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.262607 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.284780 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.411564 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.413125 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.418783 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.425122 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.434422 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.444288 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:49.466061 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:10:49.526858 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.535892 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.540030 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.549793 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.549860 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:49.557698 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:49.558856 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:10:49.559430 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
I20240503 02:10:49.606189 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.609921 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.610786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.613535 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.615661 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:49.619158 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.851238 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.853765 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.866557 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.867734 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.868337 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:49.880831 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:49.944334 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
I20240503 02:10:49.970477 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.974681 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:49.976876 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.987052 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:49.989145 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:49.990403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.027546 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.031879 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.032119 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.034363 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.036583 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.040143 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:50.138861 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:10:50.316655 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.326259 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.328015 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.328375 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:50.328635 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.330279 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:50.345448 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:10:50.388392 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.388568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.390766 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.391824 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.400574 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:50.411275 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.489504 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.490998 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.493494 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.502588 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.513610 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:50.519567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.640563 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.642650 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.642859 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:50.645423 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.651418 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.667120 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.797010 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.802943 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.805006 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.812564 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:50.812861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.827930 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:50.886113 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
I20240503 02:10:50.910163 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.913846 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.918269 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:50.925510 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:50.926038 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:50.929183 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:50.991612 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:10:51.001181 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.001433 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.005244 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:51.012939 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.019312 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.019688 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.240125 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.242028 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.242789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.245178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.255167 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.259948 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:51.297228 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240503 02:10:51.353965 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.355702 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.364485 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.366868 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:51.385484 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.389709 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.399480 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:51.399782 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.401373 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.408769 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.418782 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.431135 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.703507 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.710531 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.714413 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.720343 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.728888 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.729353 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:51.753055 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.755720 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.759078 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.771101 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.773380 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.794492 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:51.871271 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.874969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.877082 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:51.878203 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:51.882159 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:51.889434 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.058605 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:52.063885 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.068569 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.073941 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.075243 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:52.080552 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:10:52.090963 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:52.131243 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240503 02:10:52.156591 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240503 02:10:52.182003 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.186909 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.188254 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.201378 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.201915 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.208771 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:52.282902 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.289507 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.291891 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.297729 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.304010 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.312224 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:52.387815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.389889 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.394039 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.395220 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.396642 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.405396 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:52.523964 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:10:52.557880 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:10:52.600989 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.601580 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.602409 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:52.620347 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.621105 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.622534 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.718928 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.722380 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.723958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:52.729496 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.730871 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.737313 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.738763 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.748713 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.751400 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:52.774255 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:52.775624 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:52.777371 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:10:52.809662 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:10:53.065327 4734 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:53.065445 4734 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:10:53.065869 4734 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:10:53.068260 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:10:53.068521 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:10:53.068579 13103 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:53.068847 4934 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:53.076705 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.079093 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.089433 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.101954 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.113794 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.114271 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.115980 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.126019 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.129046 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.141408 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.156816 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.157496 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.236594 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.239781 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.242441 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.245270 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.253192 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.257747 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.428949 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.429530 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.432463 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.436216 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.442945 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.453351 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:53.491909 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240503 02:10:53.555648 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.559909 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.561800 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.563902 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.568744 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.586081 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.650123 3433 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:10:53.650244 3433 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:10:53.650651 3433 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:10:53.651212 14738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:10:53.652468 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:10:53.652623 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:10:53.652686 14738 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:10:53.652829 3433 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:10:53.669905 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.670951 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.671147 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.674346 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.680368 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.702279 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.762332 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.768065 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.769966 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:53.771824 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:53.776044 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.776935 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:53.940332 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240503 02:10:53.988041 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.992369 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:53.997212 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:53.997373 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.009732 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.017905 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.098163 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.099277 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.102000 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.102696 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.119016 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.131491 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.168370 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.168987 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.172434 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.172526 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.184952 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.198117 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.423686 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.424966 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.428936 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.439194 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.439313 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.440073 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.524083 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.527729 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.528632 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.536279 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.537925 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.539440 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.627568 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.631044 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.632146 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.635077 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.639901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.641785 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:54.660153 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240503 02:10:54.672147 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:10:54.672832 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:10:54.714947 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240503 02:10:54.736171 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.738628 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.740382 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.748142 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.761327 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.764315 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.942663 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:54.943931 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.944516 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:54.946347 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:54.947060 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:54.947288 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.047930 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:55.048933 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:10:55.056494 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.056684 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.067868 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.079409 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.079663 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.082475 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.092698 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.095875 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.098834 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.109401 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.114032 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:55.126259 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:10:55.427078 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.429270 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.432722 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.434003 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.445870 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.447942 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.483367 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.485751 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.485931 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.488370 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.491178 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.511380 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.577669 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.579279 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.583521 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.583634 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.594137 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.610442 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.799706 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.800184 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.800192 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.807940 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.810950 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:55.822758 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.911329 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.912528 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:55.916879 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.917886 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:55.921525 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:55.929003 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.011487 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.016233 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.019876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.020457 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.023715 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.036046 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:56.101329 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240503 02:10:56.122409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.122758 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.125286 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.136656 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.137797 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.138669 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.324851 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.325278 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.329121 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.329520 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.339851 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.346505 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.426575 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.432868 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.440044 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.444373 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:10:56.450326 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240503 02:10:56.452361 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.453797 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:56.479815 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:10:56.500831 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.501283 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.510442 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.518375 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.523939 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.530330 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.756827 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.758576 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.758647 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.760202 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.773486 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.780615 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.865623 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.874248 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.882797 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.886386 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.888285 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.892594 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.911931 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.921554 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:56.939311 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:56.947206 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:56.954823 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:56.957469 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.233688 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.235492 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.242874 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:57.244036 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240503 02:10:57.250777 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.254750 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.271837 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:57.283851 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:10:57.300362 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.302379 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.316589 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.320358 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.335259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.340273 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:57.385015 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240503 02:10:57.385178 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:10:57.429832 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.430317 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.430588 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.433351 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:57.433738 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.444077 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.623590 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.625891 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.626286 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.635649 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.637077 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.637519 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:57.647919 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:10:57.651162 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:10:57.739653 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.742159 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.744731 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.745553 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.756177 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.765930 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:57.841652 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.852545 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.857100 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.859901 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:57.860177 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.863806 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:57.954663 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.955858 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.957195 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:57.960178 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:57.960707 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:57.966346 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.160851 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.164247 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.166576 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.166728 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:58.172185 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.176261 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.280355 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.281464 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.294553 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.301072 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.304181 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.312409 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.314312 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:58.316511 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.321041 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.328430 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.346158 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.360137 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:58.488185 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240503 02:10:58.647840 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.652281 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.659462 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.662688 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.664327 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:58.677832 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.678237 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.693387 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.698230 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.702625 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.714658 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:58.715811 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.801527 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.801973 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.805294 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.807019 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:58.808714 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.816543 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.990641 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.990711 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:58.994550 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:58.995923 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:58.997637 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:59.000214 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.105445 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.105698 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.109361 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.109905 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.117977 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.129863 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:59.154058 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240503 02:10:59.211395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.211728 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.216733 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.217466 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.218398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.223912 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:59.309854 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.315976 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.316319 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:59.316453 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.328248 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.328601 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.526775 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.529579 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.529614 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.530179 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.534256 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.556396 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:10:59.578886 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240503 02:10:59.633457 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.634727 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.638355 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.645714 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.656301 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:59.679208 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.683094 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.688970 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:10:59.698249 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.700996 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.704836 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:10:59.706243 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:10:59.763325 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:10:59.802793 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:10:59.803357 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:10:59.987190 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.989640 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.994642 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:10:59.996639 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:10:59.997282 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.000085 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.055707 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.057650 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.064530 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.065987 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.073370 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:00.074106 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240503 02:11:00.074144 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.165364 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.168284 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.174482 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.175241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.182852 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.195503 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.317907 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.324079 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.324384 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.325668 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.332587 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.344069 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:11:00.428966 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:11:00.480168 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.482405 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.493062 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.494138 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.497254 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.512240 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.590077 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.591722 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.597965 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.600495 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.603178 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.623828 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.661134 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.662310 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.662410 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.666952 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.678022 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:00.686805 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.945616 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:00.947196 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.947511 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:00.952186 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.979113 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:00.981946 6338 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:11:00.982059 6338 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:00.982445 6338 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:11:00.983083 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:11:00.984071 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:11:00.984134 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:11:00.984323 6338 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:11:00.993561 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.008674 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.009222 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.016681 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.018926 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.030977 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.050031 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.112118 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.113986 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.115020 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.115864 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.117296 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:11:01.137746 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240503 02:11:01.142776 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.238981 6350 raft_consensus.cc:482] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:11:01.239092 6350 raft_consensus.cc:504] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } }
I20240503 02:11:01.239461 6350 leader_election.cc:288] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447)
W20240503 02:11:01.240463 13103 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
W20240503 02:11:01.240553 13105 leader_election.cc:334] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:11:01.240619 13105 leader_election.cc:302] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03ced8676740485e86ebe292f8ee4175; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:11:01.240736 6350 raft_consensus.cc:2726] T b4b734308d3b42958e3a2830a83d44e2 P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:11:01.333753 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.333810 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.336145 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.336514 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.337854 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.346180 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.431941 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.433395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.435521 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.440183 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.440735 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.442636 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:01.517499 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 02:11:01.534219 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.540010 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.541575 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.545078 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.551928 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.552891 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.641674 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.641922 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.644400 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.655133 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.656774 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:01.662772 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240503 02:11:01.663921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.848035 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.849776 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.857954 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:01.867872 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.868726 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:01.877158 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.974709 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.985998 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.986522 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:01.990990 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.993839 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:01.997184 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.008651 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.011161 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:11:02.011776 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:11:02.014575 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.020162 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.031922 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.044302 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:02.194846 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240503 02:11:02.337358 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.340627 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.341624 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.344729 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.345091 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.350937 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.397150 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.397573 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.400234 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.408282 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.410379 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.410789 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:02.447625 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240503 02:11:02.486707 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.489243 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.491480 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.505043 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.508268 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.508885 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:11:02.602643 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:11:02.672653 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:11:02.673600 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:11:02.709726 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.712030 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.712577 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.721922 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.722200 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.722296 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.811872 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.813519 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.815618 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.816123 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.826725 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.831880 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:02.924901 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:02.925045 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.928220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:02.929489 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.936421 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:02.942602 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.020893 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.023182 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.024185 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.025859 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.027864 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.041007 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.228193 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.231226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.235155 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.237746 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.239118 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.255357 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.347467 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.349355 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.360321 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.362262 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.362320 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.362852 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.380735 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.407630 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.407701 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.407727 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.410255 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.412680 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:03.493098 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240503 02:11:03.674757 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.681807 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.685869 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.689585 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.693958 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.698685 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.750948 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.752054 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.754921 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.759915 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.763072 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.771378 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.851415 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.852978 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.854733 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.856506 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.862885 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:03.887359 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:03.998888 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:03.999141 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:03.999145 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.001152 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.007409 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.007810 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:04.130820 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240503 02:11:04.171818 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.171847 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.173403 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.176309 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.183408 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.195041 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:04.273631 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.275028 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.277256 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.292706 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.293220 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.293625 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:04.346956 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.349093 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.350534 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.350795 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.356837 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.369288 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:04.491595 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:11:04.599710 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.601433 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.603340 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.614645 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:04.622548 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.631582 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:04.632519 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240503 02:11:04.701076 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.702904 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.705745 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.709568 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.710938 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:04.719537 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240503 02:11:04.733191 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:04.759047 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.767362 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:04.767467 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:04.768325 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.775025 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:04.784333 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:04.872932 6338 raft_consensus.cc:482] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9e39ab9b64544988b968015fe9fa06e6)
I20240503 02:11:04.873054 6338 raft_consensus.cc:504] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:04.873543 6338 leader_election.cc:288] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055), d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697)
W20240503 02:11:04.873932 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:11:04.874167 12711 leader_election.cc:341] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Not found: Tablet not found: 662782d61c6f472baab4d613b7e6f5cd
W20240503 02:11:04.874511 12712 leader_election.cc:334] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:11:04.874568 12712 leader_election.cc:302] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbba999bf95446a68b6586608b33ee16; no voters: 9e39ab9b64544988b968015fe9fa06e6, d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:11:04.874687 6338 raft_consensus.cc:2726] T 662782d61c6f472baab4d613b7e6f5cd P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:11:05.021726 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.030961 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.034632 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.035825 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.040819 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.041800 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:11:05.085515 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:11:05.114913 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.116468 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.117080 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.120874 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.126992 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.156759 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.230991 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:05.233422 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:11:05.233474 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.234980 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.237061 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.247429 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.260984 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.354795 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.356057 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.356324 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.359460 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.360785 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.380288 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.546487 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.548516 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.549206 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.551875 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.572309 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.574878 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.660815 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.662191 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.665830 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.669111 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.680652 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.682240 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.722599 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.723273 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.729128 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:05.731081 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.733304 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:05.748286 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:05.993335 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:05.999426 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.002404 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.005424 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.012181 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.013984 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.074791 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.079352 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.084766 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.093750 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.096155 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.097076 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.158540 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.160904 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.163604 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.171032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.173687 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.179708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:06.284544 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240503 02:11:06.389245 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.392320 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.395078 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.395735 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.397472 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.415974 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.490240 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.491042 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.493897 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.495255 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.496472 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.512014 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.593528 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.594282 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.597419 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.597425 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.609807 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.620224 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:06.698287 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.703388 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:06.704483 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240503 02:11:06.705463 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:11:06.705533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.706517 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.707930 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:06.708487 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:11:06.709483 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.904557 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.909395 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.911515 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:06.915971 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:06.916656 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:06.922466 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.026022 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.028365 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.037869 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.038630 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.039486 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.059577 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.066921 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.067221 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.073038 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.073086 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.075119 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:07.101305 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240503 02:11:07.110033 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:07.112334 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240503 02:11:07.403071 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.410487 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.416469 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.419858 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.421742 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.441141 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.446501 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.473767 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.478152 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.480928 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.488083 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.491710 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.573762 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.576275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.577631 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.581398 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.583911 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.602316 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:07.682842 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:11:07.683413 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:11:07.749013 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.750501 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:07.752081 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:11:07.752861 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.765434 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.766175 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.771958 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.882910 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.886399 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.888032 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:07.892355 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:07.898144 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:07.898947 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.992177 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.997288 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:07.998689 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.002605 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.010586 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.023581 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.082682 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.089589 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.090444 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.090742 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.093107 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.094930 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.312716 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.315490 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.316768 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.317662 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.329361 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.335546 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.418888 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.426189 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.426546 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.430284 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.440971 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.445297 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.456115 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.458452 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.485018 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.497131 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.500844 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.500952 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.775424 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.781723 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.782254 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.786113 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.799937 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.801295 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.836776 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.845160 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:08.845746 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.852566 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.855901 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.856521 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:08.872300 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240503 02:11:08.942510 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.944551 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.945236 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:08.945950 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:08.947916 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:08.948782 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.008870 3433 raft_consensus.cc:482] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 558837c679f246dcb3241d66a37d70f1)
I20240503 02:11:09.008986 3433 raft_consensus.cc:504] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "558837c679f246dcb3241d66a37d70f1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41447 } } peers { permanent_uuid: "754fc04ea9914073a56c0cf164f63240" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 37171 } } peers { permanent_uuid: "9e39ab9b64544988b968015fe9fa06e6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37055 } }
I20240503 02:11:09.009450 3433 leader_election.cc:288] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447), 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055)
W20240503 02:11:09.009943 14736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:11:09.010583 14738 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:11:09.010663 14736 leader_election.cc:334] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:11:09.010722 14736 leader_election.cc:302] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 754fc04ea9914073a56c0cf164f63240; no voters: 558837c679f246dcb3241d66a37d70f1, 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:11:09.010840 3433 raft_consensus.cc:2726] T 86e314466c01488bb661c596c405db51 P 754fc04ea9914073a56c0cf164f63240 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:11:09.098930 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.100868 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.102509 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.106108 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.115229 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.128624 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:09.248284 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.248605 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.249861 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.260239 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.265587 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.266084 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:09.274652 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:11:09.309747 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240503 02:11:09.351959 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.355592 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:09.356504 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.356554 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.357820 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.364182 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.444651 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.447542 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.451936 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.452244 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:09.453428 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.454969 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:09.541847 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
I20240503 02:11:09.657449 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.658849 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.659493 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.664853 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.668036 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:09.674353 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:11:09.686755 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
I20240503 02:11:09.776813 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.780066 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.783668 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.783684 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.791647 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.796702 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.801247 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:09.810197 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:09.811444 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:09.812726 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.832019 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:09.835579 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:10.063737 12710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:11:10.154821 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.159797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.160207 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.161639 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.162853 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.168632 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.169549 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.180742 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:10.190459 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:11:10.193977 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:10.204345 12972 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:11:10.209362 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.214668 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.218883 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.299425 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.303498 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.313292 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.313537 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.314167 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.332528 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.505676 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.511078 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.511412 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.515552 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.520332 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.540441 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.614238 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.614938 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.614995 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.616120 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.624498 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.641230 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.716414 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.719494 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.721516 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.725638 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.728790 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:10.736946 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.823199 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.824901 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.826690 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:10.826941 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:10.833894 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:10.844938 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.019261 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.026876 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.030366 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.031514 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.033288 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.038187 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.145226 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.147884 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.153164 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.166697 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.167420 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.174708 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.176730 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.177232 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.184937 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.188262 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.193403 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.217271 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.529902 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.533696 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.537047 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.546532 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.558373 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.559214 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.559916 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.560432 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.566890 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.580328 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.586675 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.587543 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.681073 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.681275 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.685068 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.702301 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:11.703940 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:11.709414 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:11.722224 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240503 02:11:11.838554 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240503 02:11:11.887694 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.891331 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.891794 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:11.900298 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:11.900981 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:11:11.903028 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240503 02:11:11.917217 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:11.936067 13103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:11:11.936980 13103 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:11:12.002383 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.006786 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.007396 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.009477 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.012987 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.014372 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:11:12.035571 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240503 02:11:12.113320 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.113498 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.116214 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.120970 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.122951 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.124418 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.216816 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.219296 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.222165 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.230950 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.235783 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.241695 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.423328 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.424839 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.428354 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.435452 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.438351 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.448323 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:12.519873 12710 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:11:12.543277 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.548005 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.548871 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.559228 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.560667 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.571290 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.571748 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.582831 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.589717 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.592756 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.600466 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.611528 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:11:12.821247 12974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:11:12.914487 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.917730 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.933080 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.933568 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.935226 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.937119 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.948047 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:12.948899 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.956528 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.967419 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:12.967799 10803 raft_consensus.cc:998] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.609s)
I20240503 02:11:12.968138 10803 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:11:12.970034 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:12.972679 12774 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:11:12.973096 10803 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:12.988555 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:12.994997 10219 raft_consensus.cc:998] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.340s)
I20240503 02:11:12.995314 10219 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } }
I20240503 02:11:12.998924 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:11:12.999310 10219 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:13.007329 12907 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:11:13.009048 10219 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:11:13.009423 10803 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:11:13.010046 12628 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a reported cstate change: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } }
I20240503 02:11:13.011744 12774 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:11:13.012163 12626 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 7 to 8, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
W20240503 02:11:13.018251 12619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:11:13.021875 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet d86b733f38fd4f08bd40865b99e2e7b2: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:11:13.023327 13169 consensus_queue.cc:237] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:13.024528 12907 raft_consensus.cc:1261] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Refusing update from remote peer 03ced8676740485e86ebe292f8ee4175: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:11:13.024549 13037 consensus_queue.cc:237] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
W20240503 02:11:13.024688 13104 consensus_peers.cc:485] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04582c3f86d744ccbc96eec50a10994f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:13.025031 11019 consensus_queue.cc:1035] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:11:13.026162 12774 raft_consensus.cc:1261] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:11:13.026337 12973 consensus_peers.cc:485] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d86b733f38fd4f08bd40865b99e2e7b2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:13.027082 11027 consensus_queue.cc:1035] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:11:13.027566 11019 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:13.028051 12907 raft_consensus.cc:2932] T 04582c3f86d744ccbc96eec50a10994f P 96b8f7d7c1ce4e2cad5af6caabfd8c4a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:13.030360 12633 catalog_manager.cc:5484] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 8 to 9, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "03ced8676740485e86ebe292f8ee4175" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:13.030982 11016 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:13.031944 12620 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 04582c3f86d744ccbc96eec50a10994f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:11:13.033572 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d86b733f38fd4f08bd40865b99e2e7b2 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:11:13.033875 12774 raft_consensus.cc:2932] T d86b733f38fd4f08bd40865b99e2e7b2 P bbba999bf95446a68b6586608b33ee16 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:13.034142 12626 catalog_manager.cc:5484] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 8 to 9, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:13.080396 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.082976 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.097812 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.098201 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.100028 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.109324 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.278762 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.283949 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.288358 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.291193 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.292706 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.303025 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.354235 6338 raft_consensus.cc:998] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Attempting to remove follower 558837c679f246dcb3241d66a37d70f1 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 558837c679f246dcb3241d66a37d70f1 for more than 300 seconds (300.374s)
I20240503 02:11:13.354620 6338 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } }
I20240503 02:11:13.355787 13037 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:11:13.356174 6338 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:11:13.357784 11127 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:11:13.358541 13037 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:11:13.360342 12626 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 10 to 11, VOTER 558837c679f246dcb3241d66a37d70f1 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } }
I20240503 02:11:13.366842 12774 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:13.368247 13037 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Refusing update from remote peer bbba999bf95446a68b6586608b33ee16: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:11:13.368676 6338 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:11:13.370460 6338 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
W20240503 02:11:13.371716 12711 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 -> Peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157): Couldn't send request to peer 03ced8676740485e86ebe292f8ee4175. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:13.372383 12619 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:11:13.372454 13037 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:13.373018 12626 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 reported cstate change: config changed from index 11 to 12, VOTER 03ced8676740485e86ebe292f8ee4175 (127.0.99.132) added. New cstate: current_term: 2 leader_uuid: "bbba999bf95446a68b6586608b33ee16" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:13.402434 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.404019 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.404112 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.406201 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.406211 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.412021 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.496285 11199 ts_tablet_manager.cc:902] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Initiating tablet copy from peer 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:11:13.504027 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.505556 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.506208 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.506649 11199 tablet_copy_client.cc:308] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36157
I20240503 02:11:13.509256 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.511904 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.512318 13189 tablet_copy_service.cc:140] P 03ced8676740485e86ebe292f8ee4175: Received BeginTabletCopySession request for tablet 04582c3f86d744ccbc96eec50a10994f from peer d23425fa2b5848c8b1a2b737d75e69eb ({username='slave'} at 127.0.99.134:55785)
I20240503 02:11:13.512423 13189 tablet_copy_service.cc:161] P 03ced8676740485e86ebe292f8ee4175: Beginning new tablet copy session on tablet 04582c3f86d744ccbc96eec50a10994f from peer d23425fa2b5848c8b1a2b737d75e69eb at {username='slave'} at 127.0.99.134:55785: session id = d23425fa2b5848c8b1a2b737d75e69eb-04582c3f86d744ccbc96eec50a10994f
I20240503 02:11:13.513538 13189 tablet_copy_source_session.cc:205] T 04582c3f86d744ccbc96eec50a10994f P 03ced8676740485e86ebe292f8ee4175: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:11:13.514544 11199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04582c3f86d744ccbc96eec50a10994f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:13.516753 11199 tablet_copy_client.cc:783] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 5 data blocks...
I20240503 02:11:13.522190 11199 tablet_copy_client.cc:647] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:13.523324 11199 tablet_copy_client.cc:523] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:13.524704 11199 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap starting.
I20240503 02:11:13.528342 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.535459 11209 ts_tablet_manager.cc:902] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer 22b582a5f4774861a5619c51198d39c5 (127.0.99.131:43679)
I20240503 02:11:13.535988 11209 tablet_copy_client.cc:308] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:43679
I20240503 02:11:13.536468 13055 tablet_copy_service.cc:140] P 22b582a5f4774861a5619c51198d39c5: Received BeginTabletCopySession request for tablet d86b733f38fd4f08bd40865b99e2e7b2 from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:57713)
I20240503 02:11:13.536576 13055 tablet_copy_service.cc:161] P 22b582a5f4774861a5619c51198d39c5: Beginning new tablet copy session on tablet d86b733f38fd4f08bd40865b99e2e7b2 from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:57713: session id = 03ced8676740485e86ebe292f8ee4175-d86b733f38fd4f08bd40865b99e2e7b2
I20240503 02:11:13.537565 13055 tablet_copy_source_session.cc:205] T d86b733f38fd4f08bd40865b99e2e7b2 P 22b582a5f4774861a5619c51198d39c5: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:11:13.538528 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86b733f38fd4f08bd40865b99e2e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:13.540742 11209 tablet_copy_client.cc:783] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:11:13.545161 11209 tablet_copy_client.cc:647] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:13.547055 11209 tablet_copy_client.cc:523] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:13.548522 11209 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:11:13.552908 11199 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:13.553349 11199 tablet_bootstrap.cc:492] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Bootstrap complete.
I20240503 02:11:13.553601 11199 ts_tablet_manager.cc:1367] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20240503 02:11:13.554136 11199 raft_consensus.cc:348] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:13.554338 11199 raft_consensus.cc:729] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23425fa2b5848c8b1a2b737d75e69eb, State: Initialized, Role: FOLLOWER
I20240503 02:11:13.554576 11199 consensus_queue.cc:260] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "96b8f7d7c1ce4e2cad5af6caabfd8c4a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 43951 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:13.554970 11199 ts_tablet_manager.cc:1398] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:11:13.555454 13189 tablet_copy_service.cc:342] P 03ced8676740485e86ebe292f8ee4175: Request end of tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-04582c3f86d744ccbc96eec50a10994f received from {username='slave'} at 127.0.99.134:55785
I20240503 02:11:13.555517 13189 tablet_copy_service.cc:433] P 03ced8676740485e86ebe292f8ee4175: ending tablet copy session d23425fa2b5848c8b1a2b737d75e69eb-04582c3f86d744ccbc96eec50a10994f on tablet 04582c3f86d744ccbc96eec50a10994f with peer d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:11:13.558594 11209 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:13.558853 11209 tablet_bootstrap.cc:492] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:11:13.559019 11209 ts_tablet_manager.cc:1367] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20240503 02:11:13.559365 11209 raft_consensus.cc:348] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:13.559458 11209 raft_consensus.cc:729] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:11:13.559815 11209 consensus_queue.cc:260] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:13.560724 11209 ts_tablet_manager.cc:1398] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:11:13.561378 13055 tablet_copy_service.cc:342] P 22b582a5f4774861a5619c51198d39c5: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-d86b733f38fd4f08bd40865b99e2e7b2 received from {username='slave'} at 127.0.99.132:57713
I20240503 02:11:13.561457 13055 tablet_copy_service.cc:433] P 22b582a5f4774861a5619c51198d39c5: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-d86b733f38fd4f08bd40865b99e2e7b2 on tablet d86b733f38fd4f08bd40865b99e2e7b2 with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:11:13.616354 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2): 4838 bytes on disk
I20240503 02:11:13.616865 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(d86b733f38fd4f08bd40865b99e2e7b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 02:11:13.621948 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.622259 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.623062 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.624063 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.637204 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.638029 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.717667 14718 maintenance_manager.cc:419] P d23425fa2b5848c8b1a2b737d75e69eb: Scheduling UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f): 4583 bytes on disk
I20240503 02:11:13.718173 14608 maintenance_manager.cc:643] P d23425fa2b5848c8b1a2b737d75e69eb: UndoDeltaBlockGCOp(04582c3f86d744ccbc96eec50a10994f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 02:11:13.842152 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:13.845533 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:13.849054 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.849046 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.856199 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.857648 11209 ts_tablet_manager.cc:902] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Initiating tablet copy from peer bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445)
I20240503 02:11:13.858028 11209 tablet_copy_client.cc:308] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33445
I20240503 02:11:13.858518 12796 tablet_copy_service.cc:140] P bbba999bf95446a68b6586608b33ee16: Received BeginTabletCopySession request for tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 03ced8676740485e86ebe292f8ee4175 ({username='slave'} at 127.0.99.132:35847)
I20240503 02:11:13.858614 12796 tablet_copy_service.cc:161] P bbba999bf95446a68b6586608b33ee16: Beginning new tablet copy session on tablet ae1f9c8efd2040d6842c6b9f7c6173aa from peer 03ced8676740485e86ebe292f8ee4175 at {username='slave'} at 127.0.99.132:35847: session id = 03ced8676740485e86ebe292f8ee4175-ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:11:13.859988 12796 tablet_copy_source_session.cc:205] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:11:13.864006 11209 ts_tablet_manager.cc:1875] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:11:13.866387 11209 ts_tablet_manager.cc:1888] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:11:13.866546 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1f9c8efd2040d6842c6b9f7c6173aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:13.867552 11209 tablet_copy_client.cc:783] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 5 data blocks...
I20240503 02:11:13.881989 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:13.882756 11209 tablet_copy_client.cc:647] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet copy: Starting download of 1 WAL segments...
W20240503 02:11:13.885457 12972 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240503 02:11:13.893180 11209 tablet_copy_client.cc:523] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:13.894421 11209 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Bootstrap starting.
I20240503 02:11:13.902714 11209 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:13.902942 11209 tablet_bootstrap.cc:492] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Bootstrap complete.
I20240503 02:11:13.903105 11209 ts_tablet_manager.cc:1367] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:11:13.903475 11209 raft_consensus.cc:348] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:13.903571 11209 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ced8676740485e86ebe292f8ee4175, State: Initialized, Role: FOLLOWER
I20240503 02:11:13.903735 11209 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:13.904006 11209 ts_tablet_manager.cc:1398] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:11:13.904487 12796 tablet_copy_service.cc:342] P bbba999bf95446a68b6586608b33ee16: Request end of tablet copy session 03ced8676740485e86ebe292f8ee4175-ae1f9c8efd2040d6842c6b9f7c6173aa received from {username='slave'} at 127.0.99.132:35847
I20240503 02:11:13.904564 12796 tablet_copy_service.cc:433] P bbba999bf95446a68b6586608b33ee16: ending tablet copy session 03ced8676740485e86ebe292f8ee4175-ae1f9c8efd2040d6842c6b9f7c6173aa on tablet ae1f9c8efd2040d6842c6b9f7c6173aa with peer 03ced8676740485e86ebe292f8ee4175
I20240503 02:11:13.953042 14670 raft_consensus.cc:1203] T 04582c3f86d744ccbc96eec50a10994f P d23425fa2b5848c8b1a2b737d75e69eb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:11:13.970407 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.980899 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.985555 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:13.989125 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:14.008716 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.017450 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.032248 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.033751 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.035111 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.040175 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:14.044495 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.060770 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.069474 13167 raft_consensus.cc:1203] T d86b733f38fd4f08bd40865b99e2e7b2 P 03ced8676740485e86ebe292f8ee4175 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 02:11:14.119266 13216 maintenance_manager.cc:419] P 03ced8676740485e86ebe292f8ee4175: Scheduling UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa): 4583 bytes on disk
I20240503 02:11:14.119688 13106 maintenance_manager.cc:643] P 03ced8676740485e86ebe292f8ee4175: UndoDeltaBlockGCOp(ae1f9c8efd2040d6842c6b9f7c6173aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240503 02:11:14.215564 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04582c3f86d744ccbc96eec50a10994f: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:11:14.324321 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.324626 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.324797 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.326500 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.345225 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:14.348162 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:11:14.387627 12974 consensus_peers.cc:485] T 1c464b6e30bb414bbe70ed030c5f481c P 22b582a5f4774861a5619c51198d39c5 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20240503 02:11:14.394798 12712 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): Couldn't send request to peer 9e39ab9b64544988b968015fe9fa06e6. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
I20240503 02:11:14.399919 13167 raft_consensus.cc:1203] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 02:11:14.404534 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.405817 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.406685 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.408944 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:14.417042 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.418833 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.454752 12774 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa"
dest_uuid: "bbba999bf95446a68b6586608b33ee16"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38290
I20240503 02:11:14.454877 12774 raft_consensus.cc:595] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:11:14.513772 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.516300 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.516606 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.520545 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.523550 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:11:14.540026 12710 consensus_peers.cc:485] T 0f3bdd27c750447a87691c0f57a70ee1 P bbba999bf95446a68b6586608b33ee16 -> Peer 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): Couldn't send request to peer 558837c679f246dcb3241d66a37d70f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240503 02:11:14.549136 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.630296 13017 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.632725 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.636233 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.647342 12754 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:14.647639 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.648928 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.699666 14987 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:00:57.348663 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.349115 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.353610 14990 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:00:58.417448 14990 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:00:58.982000 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.982240 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.982347 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.982420 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.982468 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.982515 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.982579 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.982650 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.982697 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.982740 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.982786 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.982831 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.982882 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.982957 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.983007 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
I20240503 02:00:59.053251 14990 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:00:59.111048 14990 rebalancer_tool.cc:1074] tablet 063a7f2ed28a455697b000f52a4343a3: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:00:59.168080 14990 rebalancer_tool.cc:1074] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:00:59.248392 14990 rebalancer_tool.cc:1074] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:00:59.278390 14990 rebalancer_tool.cc:1084] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Illegal state: tablet a105660c12094c71b3a4b80e661d9308: replica 558837c679f246dcb3241d66a37d70f1 is already present
I20240503 02:00:59.278522 14990 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:59.696657 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.696735 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.696813 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.696887 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.697225 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.697288 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.697333 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.697432 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.697479 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.697525 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.697603 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.697649 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.697695 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.697739 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.697784 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.697829 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.697876 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.697923 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.922179 14990 rebalancer_tool.cc:1074] tablet 490776bd408a48a9bcbbc7a501baa8ce: 'bbba999bf95446a68b6586608b33ee16' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:01:00.013384 14990 rebalancer_tool.cc:1074] tablet 662782d61c6f472baab4d613b7e6f5cd: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:01:00.185456 14990 rebalancer_tool.cc:1074] tablet 8108933516894d4190ef9a0cf6063be8: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:00.257519 14990 rebalancer_tool.cc:1074] tablet 04582c3f86d744ccbc96eec50a10994f: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:01:00.457106 14990 rebalancer_tool.cc:1074] tablet 6ad545d33bbc40458c75ed7c90d92ba7: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:01:00.507200 14990 rebalancer_tool.cc:1084] tablet 613074d3c4514bd5b64a06c59af09a56: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move ignored: Illegal state: tablet 613074d3c4514bd5b64a06c59af09a56: replica d23425fa2b5848c8b1a2b737d75e69eb is already present
I20240503 02:01:00.507275 14990 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.025421 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.025554 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.025635 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.041276 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.041366 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.041817 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.041927 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.041970 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.042007 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.042047 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.042110 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.042147 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.042186 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.059971 14990 rebalancer_tool.cc:1074] tablet 141097686fe24c8f9956dcc1ad1d253c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:01.103282 14990 rebalancer_tool.cc:1074] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:01.179678 14990 rebalancer_tool.cc:1084] tablet 097947dce87b409ca4a57cb752fdd5e2: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Illegal state: tablet 097947dce87b409ca4a57cb752fdd5e2: replica 558837c679f246dcb3241d66a37d70f1 is already present
I20240503 02:01:01.179787 14990 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.612624 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.612746 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.612814 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.612874 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.612951 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.613003 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.613075 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.613132 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.613178 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.613261 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.613325 14990 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
I20240503 02:01:01.639406 14990 rebalancer_tool.cc:1074] tablet 613074d3c4514bd5b64a06c59af09a56: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:01.668113 14990 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:01.732587 14990 rebalancer_tool.cc:1074] tablet 86e314466c01488bb661c596c405db51: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:06.645504 14990 rebalancer_tool.cc:1136] tablet 79c5f35eb8b546de8015c5235913efce: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:06.733211 14990 rebalancer_tool.cc:1136] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:07.972632 14990 rebalancer_tool.cc:1136] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:09.836100 14990 rebalancer_tool.cc:1136] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:09.911897 14990 rebalancer_tool.cc:1136] tablet 141097686fe24c8f9956dcc1ad1d253c: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:09.984798 14990 rebalancer_tool.cc:1136] tablet 6ad545d33bbc40458c75ed7c90d92ba7: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:09.999598 14990 rebalancer_tool.cc:1136] tablet 04582c3f86d744ccbc96eec50a10994f: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:11.221350 14990 rebalancer_tool.cc:1136] tablet 063a7f2ed28a455697b000f52a4343a3: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:11.259406 14990 rebalancer_tool.cc:1136] tablet 86e314466c01488bb661c596c405db51: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:11.291905 14990 rebalancer_tool.cc:1136] tablet 490776bd408a48a9bcbbc7a501baa8ce: 'bbba999bf95446a68b6586608b33ee16' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:12.573597 14990 rebalancer_tool.cc:1074] tablet 34e88e69beb64036bb676b771dabd2a4: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:12.614301 14990 rebalancer_tool.cc:1074] tablet b4b734308d3b42958e3a2830a83d44e2: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:12.653307 14990 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
W20240503 02:01:13.236876 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:12.871265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.0.1:57132)
0503 02:01:12.884103 (+ 12838us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:12.884116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:12.884185 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.236065 (+351880us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12767}
W20240503 02:01:13.252671 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:13.238264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.0.1:57174)
0503 02:01:13.238395 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:13.238406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:13.238471 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.252580 (+ 14109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
I20240503 02:01:13.778616 14990 rebalancer_tool.cc:1123] tablet 34e88e69beb64036bb676b771dabd2a4: '22b582a5f4774861a5619c51198d39c5' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.795094 14990 rebalancer_tool.cc:1136] tablet 8108933516894d4190ef9a0cf6063be8: '22b582a5f4774861a5619c51198d39c5' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:13.814292 14990 rebalancer_tool.cc:1136] tablet 613074d3c4514bd5b64a06c59af09a56: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:13.827294 14990 rebalancer_tool.cc:1136] tablet 007feed148fb44e9a6fe9d449b6d8f9d: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:13.847707 14990 rebalancer_tool.cc:1123] tablet 662782d61c6f472baab4d613b7e6f5cd: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '9e39ab9b64544988b968015fe9fa06e6' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
I20240503 02:01:13.862125 14990 rebalancer_tool.cc:1074] tablet fdbd697a90534ff3b7a5e17780207d88: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:13.877526 14990 rebalancer_tool.cc:1074] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:13.914693 14990 rebalancer_tool.cc:1136] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:13.931408 14990 rebalancer_tool.cc:1136] tablet fdbd697a90534ff3b7a5e17780207d88: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
W20240503 02:01:14.217087 24074 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:14.092370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.0.1:48462)
0503 02:01:14.095644 (+ 3274us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:14.095657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:14.095720 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.216861 (+121141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3211,"thread_start_us":62,"threads_started":1}
I20240503 02:01:14.229214 14990 rebalancer_tool.cc:1123] tablet b4b734308d3b42958e3a2830a83d44e2: '03ced8676740485e86ebe292f8ee4175' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:11:14.690595 14990 rebalancer_tool.cc:1123] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: '22b582a5f4774861a5619c51198d39c5' -> '' move is abandoned: Remote error: Service unavailable: leader transfer in progress
I20240503 02:11:14.690673 14990 rebalancer_tool.cc:748] detected a staleness period of 600.461s
Incomplete: stalled with no progress for more than 300.000s seconds, aborting
I20240503 02:11:14.778846 12623 maintenance_manager.cc:419] P e267e0608b874bc396051ef2febea58d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1999 bytes on disk
I20240503 02:11:14.781230 12622 maintenance_manager.cc:643] P e267e0608b874bc396051ef2febea58d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"cfile_init":1,"dirs.queue_time_us":117,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"thread_start_us":119,"threads_started":2,"wal-append.queue_time_us":141}
I20240503 02:11:14.809768 11604 raft_consensus.cc:979] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: : Instructing follower 22b582a5f4774861a5619c51198d39c5 to start an election
I20240503 02:11:14.809880 11604 raft_consensus.cc:1067] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Signalling peer 22b582a5f4774861a5619c51198d39c5 to start an election
I20240503 02:11:14.810355 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa"
dest_uuid: "22b582a5f4774861a5619c51198d39c5"
from {username='slave'} at 127.0.99.129:51779
I20240503 02:11:14.810528 13037 raft_consensus.cc:482] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:14.810613 13037 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:11:14.811661 13037 raft_consensus.cc:504] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:14.812055 13037 leader_election.cc:288] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 election: Requested vote from peers bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445), 03ced8676740485e86ebe292f8ee4175 (127.0.99.132:36157)
I20240503 02:11:14.812671 12774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "bbba999bf95446a68b6586608b33ee16"
I20240503 02:11:14.812789 12774 raft_consensus.cc:3032] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:11:14.812830 12774 raft_consensus.cc:729] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbba999bf95446a68b6586608b33ee16, State: Running, Role: LEADER
I20240503 02:11:14.812980 12774 consensus_queue.cc:260] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:14.813140 12774 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:11:14.814186 12774 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 3.
I20240503 02:11:14.814640 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1f9c8efd2040d6842c6b9f7c6173aa" candidate_uuid: "22b582a5f4774861a5619c51198d39c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "03ced8676740485e86ebe292f8ee4175"
I20240503 02:11:14.814766 13167 raft_consensus.cc:3037] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:11:14.815830 13167 raft_consensus.cc:2454] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22b582a5f4774861a5619c51198d39c5 in term 3.
I20240503 02:11:14.816366 12972 leader_election.cc:302] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22b582a5f4774861a5619c51198d39c5, bbba999bf95446a68b6586608b33ee16; no voters:
I20240503 02:11:14.816547 11471 raft_consensus.cc:2781] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:11:14.816622 11471 raft_consensus.cc:686] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Becoming Leader. State: Replica: 22b582a5f4774861a5619c51198d39c5, State: Running, Role: LEADER
I20240503 02:11:14.816748 11471 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:14.819274 12626 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 reported cstate change: term changed from 2 to 3, leader changed from bbba999bf95446a68b6586608b33ee16 (127.0.99.129) to 22b582a5f4774861a5619c51198d39c5 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 } } peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:14.831882 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.834741 12754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:14.837394 13017 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:14.842710 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.848177 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:14.852066 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.936036 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.938864 13017 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:14.942411 12754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:14.948346 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:14.953001 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:14.965682 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:15.231345 13167 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:11:15.231864 10803 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:11:15.235410 12774 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:11:15.236160 11471 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbba999bf95446a68b6586608b33ee16" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:11:15.247840 12619 catalog_manager.cc:4573] TS 558837c679f246dcb3241d66a37d70f1 (127.0.99.136:41447): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04582c3f86d744ccbc96eec50a10994f: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: connect: Connection refused (error 111)
I20240503 02:11:15.253235 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:15.254567 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:15.258632 12754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:15.259243 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:15.274641 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:15.275332 13017 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:11:15.298861 12712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:11:15.327936 13037 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } }
I20240503 02:11:15.328954 13167 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:11:15.329332 11471 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:11:15.330602 10803 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:15.330996 13167 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:15.333307 12626 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 reported cstate change: config changed from index 12 to 14, VOTER bbba999bf95446a68b6586608b33ee16 (127.0.99.129) evicted. New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } }
I20240503 02:11:15.336191 12754 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with delete_type TABLET_DATA_TOMBSTONED (TS bbba999bf95446a68b6586608b33ee16 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:38182
I20240503 02:11:15.336555 11726 tablet_replica.cc:330] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: stopping tablet replica
I20240503 02:11:15.336719 11726 raft_consensus.cc:2229] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:15.336854 11726 raft_consensus.cc:2258] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:15.336892 13037 consensus_queue.cc:237] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } }
I20240503 02:11:15.337687 11726 ts_tablet_manager.cc:1875] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:11:15.338490 12973 consensus_peers.cc:485] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 -> Peer d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134:40697): Couldn't send request to peer d23425fa2b5848c8b1a2b737d75e69eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae1f9c8efd2040d6842c6b9f7c6173aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:15.338713 13167 raft_consensus.cc:1261] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Refusing update from remote peer 22b582a5f4774861a5619c51198d39c5: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 02:11:15.339087 11471 consensus_queue.cc:1035] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:11:15.340524 10803 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:15.341136 11726 ts_tablet_manager.cc:1888] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240503 02:11:15.341202 11726 log.cc:1196] T ae1f9c8efd2040d6842c6b9f7c6173aa P bbba999bf95446a68b6586608b33ee16: Deleting WAL directory at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae1f9c8efd2040d6842c6b9f7c6173aa
I20240503 02:11:15.341676 12619 catalog_manager.cc:4830] TS bbba999bf95446a68b6586608b33ee16 (127.0.99.129:33445): tablet ae1f9c8efd2040d6842c6b9f7c6173aa (table rebalance_test_table_7 [id=0c35edfb7bc44311b8759dd3674bc8a9]) successfully deleted
I20240503 02:11:15.341763 12621 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ae1f9c8efd2040d6842c6b9f7c6173aa with cas_config_opid_index 14: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:11:15.342115 13167 raft_consensus.cc:2932] T ae1f9c8efd2040d6842c6b9f7c6173aa P 03ced8676740485e86ebe292f8ee4175 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:15.342844 12626 catalog_manager.cc:5484] T ae1f9c8efd2040d6842c6b9f7c6173aa P 22b582a5f4774861a5619c51198d39c5 reported cstate change: config changed from index 14 to 15, VOTER d23425fa2b5848c8b1a2b737d75e69eb (127.0.99.134) added. New cstate: current_term: 3 leader_uuid: "22b582a5f4774861a5619c51198d39c5" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "22b582a5f4774861a5619c51198d39c5" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 43679 } } peers { permanent_uuid: "03ced8676740485e86ebe292f8ee4175" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36157 } } peers { permanent_uuid: "d23425fa2b5848c8b1a2b737d75e69eb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40697 } } }
I20240503 02:11:15.603253 13017 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:15.607434 14650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:15.607681 13149 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:15.610397 12754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:15.610950 14782 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:15.622241 12887 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:15.661792 14983 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:00:57.443277 14991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:00:57.443524 14991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:00:57.446995 14991 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:00:58.390465 14991 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:00:58.919377 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.919543 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.919715 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.919850 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.919927 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.919998 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.920048 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.920117 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.920209 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:58.920334 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.920454 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:58.920526 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:58.920596 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:58.920655 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.072721 14991 rebalancer_tool.cc:1074] tablet b4b734308d3b42958e3a2830a83d44e2: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move scheduled
I20240503 02:00:59.134845 14991 rebalancer_tool.cc:1084] tablet e6cc67c73560449e81303a948df8f9df: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 02:00:59.134960 14991 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:00:59.505820 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.505908 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.505982 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.506026 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506088 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506170 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.506238 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.506287 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506354 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.506456 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506497 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506551 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.506578 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:00:59.506621 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:00:59.506649 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:00:59.506677 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:00:59.506704 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:00:59.839617 14991 rebalancer_tool.cc:1074] tablet 86e314466c01488bb661c596c405db51: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move scheduled
I20240503 02:00:59.975126 14991 rebalancer_tool.cc:1074] tablet 007feed148fb44e9a6fe9d449b6d8f9d: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move scheduled
I20240503 02:00:59.996186 14991 rebalancer_tool.cc:1084] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move ignored: Illegal state: tablet 613074d3c4514bd5b64a06c59af09a56: replica d23425fa2b5848c8b1a2b737d75e69eb is already present
I20240503 02:00:59.996259 14991 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:00.401957 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.402115 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.402218 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.402297 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.402393 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.402444 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.402475 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.402504 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.402536 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.402570 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.402606 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.402662 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.402712 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.402740 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.402799 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:00.402837 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:00.402866 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 850edfd853f047b19d57394a1b67d1f2: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:00.402927 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:00.402962 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
I20240503 02:01:00.506491 14991 rebalancer_tool.cc:1074] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:00.740161 14991 rebalancer_tool.cc:1074] tablet 8292c06377f445849acdc22d08bd612d: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move scheduled
I20240503 02:01:00.922561 14991 rebalancer_tool.cc:1084] tablet 0f3bdd27c750447a87691c0f57a70ee1: 'bbba999bf95446a68b6586608b33ee16' -> '558837c679f246dcb3241d66a37d70f1' move ignored: Illegal state: tablet 0f3bdd27c750447a87691c0f57a70ee1: replica 558837c679f246dcb3241d66a37d70f1 is already present
I20240503 02:01:00.922642 14991 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:01:01.209811 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0c35edfb7bc44311b8759dd3674bc8a9: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.209914 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f4d2c6e338a4a099ebc28f1418baf11: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.209965 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.210059 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bf094dc6792f4586844a8fa66aa149a1: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 558837c679f246dcb3241d66a37d70f1
W20240503 02:01:01.210099 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 87f74cad3e4e48a389dd92a283273233: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.210183 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 99b49350f69a4139a63c1dd150969f7f: could not find any suitable replica to move from server 96b8f7d7c1ce4e2cad5af6caabfd8c4a to server d23425fa2b5848c8b1a2b737d75e69eb
W20240503 02:01:01.210224 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table baadc882e6ce4f52b3d62a2bdf4a4b63: could not find any suitable replica to move from server 03ced8676740485e86ebe292f8ee4175 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.210261 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8a53b815c07146bf988000c73f7e70d4: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
W20240503 02:01:01.210330 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 60828393e9d5433c908a3ba9783dd7a0: could not find any suitable replica to move from server bbba999bf95446a68b6586608b33ee16 to server 9e39ab9b64544988b968015fe9fa06e6
W20240503 02:01:01.210371 14991 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 319c91db9ef042e9a8e1ad6fb0b8cd24: could not find any suitable replica to move from server 22b582a5f4774861a5619c51198d39c5 to server 754fc04ea9914073a56c0cf164f63240
I20240503 02:01:01.228122 14991 rebalancer_tool.cc:1074] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:01.260299 14991 rebalancer_tool.cc:1074] tablet 13a95f88ac1c459ba01be0743297f696: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:01.268613 14991 rebalancer_tool.cc:1074] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:01.278687 14991 rebalancer_tool.cc:1074] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:04.443840 14991 rebalancer_tool.cc:1136] tablet 007feed148fb44e9a6fe9d449b6d8f9d: '03ced8676740485e86ebe292f8ee4175' -> '754fc04ea9914073a56c0cf164f63240' move completed: OK
I20240503 02:01:05.997678 14991 rebalancer_tool.cc:1136] tablet a105660c12094c71b3a4b80e661d9308: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:06.725121 14991 rebalancer_tool.cc:1074] tablet 063a7f2ed28a455697b000f52a4343a3: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:06.748827 14991 rebalancer_tool.cc:1136] tablet 13a95f88ac1c459ba01be0743297f696: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:07.289559 14991 rebalancer_tool.cc:1136] tablet b4b734308d3b42958e3a2830a83d44e2: 'bbba999bf95446a68b6586608b33ee16' -> '9e39ab9b64544988b968015fe9fa06e6' move completed: OK
I20240503 02:01:07.327941 14991 rebalancer_tool.cc:1136] tablet 662782d61c6f472baab4d613b7e6f5cd: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:07.372491 14991 rebalancer_tool.cc:1136] tablet 0613ddd56ed346b3b6dfa9e7c3a4335e: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:08.371943 14991 rebalancer_tool.cc:1136] tablet 613074d3c4514bd5b64a06c59af09a56: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:08.393863 14991 rebalancer_tool.cc:1136] tablet 8292c06377f445849acdc22d08bd612d: '22b582a5f4774861a5619c51198d39c5' -> 'd23425fa2b5848c8b1a2b737d75e69eb' move completed: OK
I20240503 02:01:08.438237 14991 rebalancer_tool.cc:1074] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:08.498056 14991 rebalancer_tool.cc:1074] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:08.566401 14991 rebalancer_tool.cc:1074] tablet 1c464b6e30bb414bbe70ed030c5f481c: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:08.579569 14991 rebalancer_tool.cc:1074] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:08.588064 14991 rebalancer_tool.cc:1136] tablet b39fa1cb3863414287a47a0832c8b55d: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
I20240503 02:01:09.160154 14991 rebalancer_tool.cc:1136] tablet d86b733f38fd4f08bd40865b99e2e7b2: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:10.098120 14991 rebalancer_tool.cc:1136] tablet 063a7f2ed28a455697b000f52a4343a3: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:10.113771 14991 rebalancer_tool.cc:1074] tablet e6cc67c73560449e81303a948df8f9df: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move scheduled
I20240503 02:01:10.269181 14991 rebalancer_tool.cc:1074] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:10.388391 14991 rebalancer_tool.cc:1074] tablet 34e88e69beb64036bb676b771dabd2a4: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:10.412503 14991 rebalancer_tool.cc:1136] tablet 0633596226954c439d2f671c25eebfe6: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:12.029719 14991 rebalancer_tool.cc:1136] tablet 1c464b6e30bb414bbe70ed030c5f481c: '03ced8676740485e86ebe292f8ee4175' -> '' move completed: OK
I20240503 02:01:12.077956 14991 rebalancer_tool.cc:1136] tablet fdbd697a90534ff3b7a5e17780207d88: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
I20240503 02:01:12.127310 14991 rebalancer_tool.cc:1136] tablet 86e314466c01488bb661c596c405db51: '22b582a5f4774861a5619c51198d39c5' -> '558837c679f246dcb3241d66a37d70f1' move completed: OK
I20240503 02:01:12.140650 14991 rebalancer_tool.cc:1074] tablet f5767bdfcc6249578f7c3ada3c08da6e: '03ced8676740485e86ebe292f8ee4175' -> '' move scheduled
I20240503 02:01:12.156628 14991 rebalancer_tool.cc:1074] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: 'bbba999bf95446a68b6586608b33ee16' -> '' move scheduled
I20240503 02:01:12.164080 14991 rebalancer_tool.cc:1074] tablet 8108933516894d4190ef9a0cf6063be8: '22b582a5f4774861a5619c51198d39c5' -> '' move scheduled
I20240503 02:01:12.178640 14991 rebalancer_tool.cc:1136] tablet 8108933516894d4190ef9a0cf6063be8: '22b582a5f4774861a5619c51198d39c5' -> '' move completed: OK
W20240503 02:01:13.239573 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:12.969325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37055 (local address 127.0.0.1:57142)
0503 02:01:13.004947 (+ 35622us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:13.004961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:13.005050 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:13.238877 (+233827us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35530,"thread_start_us":97,"threads_started":1}
I20240503 02:01:13.242331 14991 rebalancer_tool.cc:1123] tablet f5767bdfcc6249578f7c3ada3c08da6e: '03ced8676740485e86ebe292f8ee4175' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:01:13.250770 14991 rebalancer_tool.cc:1136] tablet e6cc67c73560449e81303a948df8f9df: '96b8f7d7c1ce4e2cad5af6caabfd8c4a' -> '' move completed: OK
I20240503 02:01:13.263278 14991 rebalancer_tool.cc:1123] tablet 34e88e69beb64036bb676b771dabd2a4: '22b582a5f4774861a5619c51198d39c5' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37055: connect: Connection refused (error 111)
W20240503 02:01:14.217669 24039 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:14.215800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:41447 (local address 127.0.0.1:48474)
0503 02:01:14.215986 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:14.215999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:14.216078 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:14.217560 (+ 1482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:41447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
W20240503 02:01:25.368274 27445 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.99.190:37619, user_credentials={real_user=slave}} blocked reactor thread for 47103.4us
I20240503 02:11:15.540063 14991 rebalancer_tool.cc:1136] tablet ae1f9c8efd2040d6842c6b9f7c6173aa: 'bbba999bf95446a68b6586608b33ee16' -> '' move completed: OK
Illegal state: tablet server 9e39ab9b64544988b968015fe9fa06e6 (127.0.99.133:37055): unacceptable health status UNAVAILABLE
E20240503 02:11:15.662169 398 external_mini_cluster.cc:943] Process with UUID 9e39ab9b64544988b968015fe9fa06e6 has crashed
E20240503 02:11:15.662271 398 external_mini_cluster.cc:943] Process with UUID 558837c679f246dcb3241d66a37d70f1 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240503 02:11:15.662922 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 12694
I20240503 02:11:15.671219 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 12825
I20240503 02:11:15.679734 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 12956
I20240503 02:11:15.688237 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 13087
I20240503 02:11:15.695848 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 14589
I20240503 02:11:15.702714 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 14720
I20240503 02:11:15.710508 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu with pid 12603
2024-05-03T02:11:15Z chronyd exiting
I20240503 02:11:15.728391 398 test_util.cc:175] -----------------------------------------------
I20240503 02:11:15.728441 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714701545747728-398-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (630011 ms)
[----------] 1 test from ConcurrentRebalancersTest (630011 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240503 02:11:15.729025 398 test_util.cc:268] Using random seed: 1007310462
I20240503 02:11:15.729728 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:15.729763 398 ts_itest-base.cc:113] --------------
I20240503 02:11:15.729774 398 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:11:15.729784 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:11:15.729794 398 ts_itest-base.cc:116] --------------
2024-05-03T02:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:15Z Disabled control of system clock
I20240503 02:11:15.737294 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46865
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:46865
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 02:11:15.842816 11782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:15.843019 11782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:15.843086 11782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:15.846043 11782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:15.846117 11782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:15.846180 11782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:15.846217 11782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:15.846259 11782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:15.849745 11782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:46865
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:11:15.850529 11782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:15.851585 11782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:15.859405 11790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:15.859537 11787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:15.859944 11788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:15.860795 11782 server_base.cc:1034] running on GCE node
I20240503 02:11:15.861232 11782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:15.862071 11782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:15.863335 11782 hybrid_clock.cc:648] HybridClock initialized: now 1714702275863307 us; error 41 us; skew 500 ppm
I20240503 02:11:15.865144 11782 webserver.cc:460] Webserver started at http://127.0.99.190:32929/ using document root <none> and password file <none>
I20240503 02:11:15.865648 11782 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:15.865705 11782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:15.865862 11782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:15.867383 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "de9a183847cf4a9c824c88a29cc48d5f"
format_stamp: "Formatted at 2024-05-03 02:11:15 on dist-test-slave-fvr5"
I20240503 02:11:15.867925 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "de9a183847cf4a9c824c88a29cc48d5f"
format_stamp: "Formatted at 2024-05-03 02:11:15 on dist-test-slave-fvr5"
I20240503 02:11:15.871018 11782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:15.873049 11796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:15.873982 11782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:15.874091 11782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "de9a183847cf4a9c824c88a29cc48d5f"
format_stamp: "Formatted at 2024-05-03 02:11:15 on dist-test-slave-fvr5"
I20240503 02:11:15.874172 11782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:15.895373 11782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:15.895960 11782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:15.896117 11782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:15.902313 11782 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:46865
I20240503 02:11:15.902326 11848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:46865 every 8 connection(s)
I20240503 02:11:15.903215 11782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:15.905643 11849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:15.910248 11849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: Bootstrap starting.
I20240503 02:11:15.911716 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu as pid 11782
I20240503 02:11:15.911857 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:15.912326 11849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:15.912997 11849 log.cc:824] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:15.914528 11849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: No bootstrap required, opened a new log
I20240503 02:11:15.916882 11849 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } }
I20240503 02:11:15.917047 11849 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:15.917124 11849 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9a183847cf4a9c824c88a29cc48d5f, State: Initialized, Role: FOLLOWER
I20240503 02:11:15.917533 11849 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } }
I20240503 02:11:15.917655 11849 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:15.917721 11849 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:15.917827 11849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:15.918486 11849 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } }
I20240503 02:11:15.918869 11849 leader_election.cc:302] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9a183847cf4a9c824c88a29cc48d5f; no voters:
I20240503 02:11:15.919127 11849 leader_election.cc:288] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:15.919540 11854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:15.919728 11854 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [term 1 LEADER]: Becoming Leader. State: Replica: de9a183847cf4a9c824c88a29cc48d5f, State: Running, Role: LEADER
I20240503 02:11:15.920056 11854 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } }
I20240503 02:11:15.920138 11849 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:15.921533 11856 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9a183847cf4a9c824c88a29cc48d5f. Latest consensus state: current_term: 1 leader_uuid: "de9a183847cf4a9c824c88a29cc48d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } } }
I20240503 02:11:15.921532 11855 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9a183847cf4a9c824c88a29cc48d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9a183847cf4a9c824c88a29cc48d5f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 46865 } } }
I20240503 02:11:15.921679 11855 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:15.921679 11856 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f [sys.catalog]: This master's current role is: LEADER
W20240503 02:11:15.924249 11869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 02:11:15.924309 11869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 02:11:15.924384 11867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:15.925096 11867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:15.929566 11867 catalog_manager.cc:1347] Generated new cluster ID: 6f3219eb899d4a5db6cd0e27c396cc9b
I20240503 02:11:15.929612 11867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:15.939430 11867 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:15.940232 11867 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:15.955128 11867 catalog_manager.cc:5857] T 00000000000000000000000000000000 P de9a183847cf4a9c824c88a29cc48d5f: Generated new TSK 0
I20240503 02:11:15.956056 11867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:15.961969 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
/tmp/dist-test-taskA9vV4k/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46865
--builtin_ntp_servers=127.0.99.148:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:11:16.068202 11873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:16.068424 11873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:16.068496 11873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:16.071372 11873 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:16.071444 11873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:16.071561 11873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:11:16.074963 11873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:46865
--never_fsync=true
--heap_profile_path=/tmp/kudu.11873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:11:16.075716 11873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:16.076769 11873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:16.084877 11879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:16.085489 11881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:16.086093 11878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:16.086639 11873 server_base.cc:1034] running on GCE node
I20240503 02:11:16.086973 11873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:16.087452 11873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:16.088590 11873 hybrid_clock.cc:648] HybridClock initialized: now 1714702276088567 us; error 41 us; skew 500 ppm
I20240503 02:11:16.090260 11873 webserver.cc:460] Webserver started at http://127.0.99.129:43145/ using document root <none> and password file <none>
I20240503 02:11:16.090795 11873 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:16.090849 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:16.091022 11873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:16.092567 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA9vV4k/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714701545747728-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d1ff95128a97401a9dddd2d807086da6"
format_stamp: "Formatted at 2024-05-03 02:11:16 on dist-test-slave-fvr5"